pssh() passh -p <(pass show network/ssh/password head -1) ssh "$@" compdef pssh=ssh
ssh
command to look for a password prompt. Moreover, if no password is required, the
password manager is still invoked. Since OpenSSH 8.4, we can use
SSH_ASKPASS and SSH_ASKPASS_REQUIRE instead:
ssh() set -o localoptions -o localtraps local passname=network/ssh/password local helper=$(mktemp) trap "command rm -f $helper" EXIT INT > $helper <<EOF #!$SHELL pass show $passname head -1 EOF chmod u+x $helper SSH_ASKPASS=$helper SSH_ASKPASS_REQUIRE=force command ssh "$@"
ssh() set -o localoptions -o localtraps local passname=network/ssh/password local helper=$(mktemp) trap "command rm -f $helper" EXIT INT > $helper <<EOF #!$SHELL if [ -k $helper ]; then oldtty=\$(stty -g) trap 'stty \$oldtty < /dev/tty 2> /dev/null' EXIT INT TERM HUP stty -echo print "\rpassword: " read password printf "\n" > /dev/tty < /dev/tty printf "%s" "\$password" else pass show $passname head -1 chmod +t $helper fi EOF chmod u+x $helper SSH_ASKPASS=$helper SSH_ASKPASS_REQUIRE=force command ssh "$@"
ssh() # Grab login information local -A details details=($ =$ (M)$ :-"$ (@f)$(command ssh -G "$@" 2>/dev/null) " :#(host hostname user) * ) local remote=$ details[host]:-details[hostname] local login=$ details[user] @$ remote # Get password name local passname case "$login" in admin@*.example.net) passname=company1/ssh/admin ;; bernat@*.example.net) passname=company1/ssh/bernat ;; backup@*.example.net) passname=company1/ssh/backup ;; esac # No password name? Just use regular SSH [[ -z $passname ]] && command ssh "$@" return $? # Invoke SSH with the helper for SSH_ASKPASS # [ ]
scp invoke our custom ssh function:
scp() set -o localoptions -o localtraps local helper=$(mktemp) trap "command rm -f $helper" EXIT INT > $helper <<EOF #!$SHELL source $ (%):-%x ssh "\$@" EOF command scp -S $helper "$@"
ssh() function body into its own script file and replace command ssh
with /usr/bin/ssh to avoid an unwanted recursive call. In this case, the
scp() function is not needed anymore.
print -l and the
zshexpn(1) manual page. details is an associative array defined
from an array alternating keys and values.
Welcome to the August 2023 report from the Reproducible Builds project!
In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.
The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. If you are interested in contributing to the project, please visit our Contribute page on our website.
serde_derive macro as a precompiled binary. As Ax Sharma writes:
The move has generated a fair amount of push back among developers who worry about its future legal and technical implications, along with a potential for supply chain attacks, should the maintainer account publishing these binaries be compromised.After intensive discussions, use of the precompiled binary was phased out.
On August 4th, Holger Levsen gave a talk at BornHack 2023 on the Danish island of Funen titled Reproducible Builds, the first ten years which promised to contain:
[ ] an overview about reproducible builds, the past, the presence and the future. How it started with a small [meeting] at DebConf13 (and before), how it grew from being a Debian effort to something many projects work on together, until in 2021 it was mentioned in an executive order of the president of the United States. (HTML slides)Holger repeated the talk later in the month at Chaos Communication Camp 2023 in Zehdenick, Germany: A video of the talk is available online, as are the HTML slides.
Just another reminder that our upcoming Reproducible Builds Summit is set to take place from October 31st November 2nd 2023 in Hamburg, Germany.
Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field.
If you re interested in joining us this year, please make sure to read the event page, the news item, or the invitation email that Mattia Rizzolo sent out, which have more details about the event and location.
We are also still looking for sponsors to support the event, so do reach out to the organizing team if you are able to help. (Also of note that PackagingCon 2023 is taking place in Berlin just before our summit, and their schedule has just been published.)
Vagrant Cascadian was interviewed on the SustainOSS podcast on reproducible builds:
Vagrant walks us through his role in the project where the aim is to ensure identical results in software builds across various machines and times, enhancing software security and creating a seamless developer experience. Discover how this mission, supported by the Software Freedom Conservancy and a broad community, is changing the face of Linux distros, Arch Linux, openSUSE, and F-Droid. They also explore the challenges of managing random elements in software, and Vagrant s vision to make reproducible builds a standard best practice that will ideally become automatic for users. Vagrant shares his work in progress and their commitment to the last mile problem.The episode is available to listen (or download) from the Sustain podcast website. As it happens, the episode was recorded at FOSSY 2023, and the video of Vagrant s talk from this conference (Breaking the Chains of Trusting Trust is now available on Archive.org:
It was also announced that Vagrant Cascadian will be presenting at the Open Source Firmware Conference in October on the topic of Reproducible Builds All The Way Down.
hello-traditional package from Debian. The entire thread can be viewed from the archive page, as can Vagrant Cascadian s reply.
In diffoscope development this month, versions 247, 248 and 249 were uploaded to Debian unstable by Chris Lamb, who also added documentation for the new specialize_as method and expanding the documentation of the existing specialize as well [ ]. In addition, Fay Stegerman added specialize_as and used it to optimise .smali comparisons when decompiling Android .apk files [ ], Felix Yan and Mattia Rizzolo corrected some typos in code comments [ , ], Greg Chabala merged the RUN commands into single layer in the package s Dockerfile [ ] thus greatly reducing the final image size. Lastly, Roland Clobus updated tool descriptions to mark that the xb-tool has moved package within Debian [ ].
In Debian, 28 reviews of Debian packages were added, 14 were updated and 13 were removed this month adding to our knowledge about identified issues. A number of issue types were added, including Chris Lamb adding a new timestamp_in_documentation_using_sphinx_zzzeeksphinx_theme toolchain issue.
In August, F-Droid added 25 new reproducible apps and saw 2 existing apps switch to reproducible builds, making 191 apps in total that are published with Reproducible Builds and using the upstream developer s signature. [ ]
Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.
arimo (modification time in build results)apptainer (random Go build identifier)arrow (fails to build on single-CPU machines)camlp (parallelism-related issue)developer (Go ordering-related issue)elementary-xfce-icon-theme (font-related problem)gegl (parallelism issue)grommunio (filesystem ordering issue)grpc (drop nondetermistic log)guile-parted (parallelism-related issue)icinga (hostname-based issue)liquid-dsp (CPU-oriented problem)memcached (package fails to build far in the future)openmpi5/openpmix (date/copyright year issue)openmpi5 (date/copyright year issue)orthanc-ohif+orthanc-volview (ordering related issue plus timestamp in a Gzip)perl-Net-DNS (package fails to build far in the future)postgis (parallelism issue)python-scipy (uses an arbitrary build path)python-trustme (package fails to build far in the future)qtbase/qmake/goldendict-ng (timestamp-related issue)qtox (date-related issue)ring (filesytem ordering related issue)scipy (1 & 2) (drop arbtirary build path and filesytem-ordering issue)snimpy (1 & 3) (fails to build on single-CPU machines as well far in the future)tango-icon-theme (font-related issue)
The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen:
reproducible-tracker.json data file. [ ]pbuilder.tgz for Debian unstable due to #1050784. [ ][ ]usrmerge. [ ][ ]armhf nodes (wbq0 and jtx1a) as down; investigation is needed. [ ]buildd.debian.org. [ ][ ]
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
eecr-relay wave shape fit, see the correction to our initial article in JACS. We strongly encourage all the users of the eecr-relay wave shape fit to upgrade, and, unfortunately, refit previously fitted data as the results might change. The other wave shape fits are not affected by the issue.
New features
In addition to this important bug fix, new possibilities have been added, including a way to make fits with partially globalparameters using the new
define-indexed-fit command, to pick the best parameters dataset-by-dataset within fit trajectories, but also a parameter space explorer trying all possible permutations of one or more sets of parameters, and the possibility to save the results of a command to a global ruby variable.
There are a lot of other new features, improvements and so on, look for the full list there.
About QSoasSELECT * FROM ( SELECT toStartOfMinute(TimeReceived) AS TimeReceived, DstAddr, SrcPort, dictGetOrDefault('protocols', 'name', Proto, '???') AS Proto, SUM(((((Bytes * SamplingRate) * 8) / 1000) / 1000) / 1000) / 60 AS Gbps, uniq(SrcAddr) AS sources, uniq(SrcCountry) AS countries FROM flows WHERE TimeReceived > now() - INTERVAL 5 MINUTE AND DstNetRole = 'customers' GROUP BY TimeReceived, DstAddr, SrcPort, Proto ) WHERE (Gbps > 1) OR ((Proto = 'UDP') AND (Gbps > 0.2)) OR ((sources > 20) AND (Gbps > 0.1)) OR ((countries > 10) AND (Gbps > 0.1)) ORDER BY TimeReceived DESC, Gbps DESC
| TimeReceived | DstAddr | SrcPort | Proto | Gbps | sources | countries |
|---|---|---|---|---|---|---|
| 2023-02-26 17:44:00 | ::ffff:203.0.113.206 |
123 | UDP | 0.102 | 109 | 13 |
| 2023-02-26 17:43:00 | ::ffff:203.0.113.206 |
123 | UDP | 0.130 | 133 | 17 |
| 2023-02-26 17:43:00 | ::ffff:203.0.113.68 |
53 | UDP | 0.129 | 364 | 63 |
| 2023-02-26 17:43:00 | ::ffff:203.0.113.206 |
123 | UDP | 0.113 | 129 | 21 |
| 2023-02-26 17:42:00 | ::ffff:203.0.113.206 |
123 | UDP | 0.139 | 50 | 14 |
| 2023-02-26 17:42:00 | ::ffff:203.0.113.206 |
123 | UDP | 0.105 | 42 | 14 |
| 2023-02-26 17:40:00 | ::ffff:203.0.113.68 |
53 | UDP | 0.121 | 340 | 65 |
65535:666 for this purpose. This is known as a
remote-triggered blackhole (RTBH) and is explained in more detail in RFC 3882.
It is also possible to blackhole the source of the attacks by leveraging
unicast Reverse Path Forwarding (uRPF) from RFC 3704, as explained in RFC 5635. However, uRPF can be a serious tax on your router resources. See
NCS5500 uRPF: Configuration and Impact on Scale for an example of the kind
of restrictions you have to expect when enabling uRPF.
On the advertising side, we can use BIRD. Here is a complete configuration
file to allow any router to collect them:
log stderr all; router id 192.0.2.1; protocol device scan time 10; protocol bgp exporter ipv4 import none; export where proto = "blackhole4"; ; ipv6 import none; export where proto = "blackhole6"; ; local as 64666; neighbor range 192.0.2.0/24 external; multihop; dynamic name "exporter"; dynamic name digits 2; graceful restart yes; graceful restart time 0; long lived graceful restart yes; long lived stale time 3600; # keep routes for 1 hour! protocol static blackhole4 ipv4; route 203.0.113.206/32 blackhole bgp_community.add((65535, 666)); ; route 203.0.113.68/32 blackhole bgp_community.add((65535, 666)); ; protocol static blackhole6 ipv6;
router static vrf public address-family ipv4 unicast 192.0.2.1/32 Null0 description "BGP blackhole" ! address-family ipv6 unicast 2001:db8::1/128 Null0 description "BGP blackhole" ! ! ! route-policy blackhole_ipv4_in_public if destination in (0.0.0.0/0 le 31) then drop endif set next-hop 192.0.2.1 done end-policy ! route-policy blackhole_ipv6_in_public if destination in (::/0 le 127) then drop endif set next-hop 2001:db8::1 done end-policy ! router bgp 12322 neighbor-group BLACKHOLE_IPV4_PUBLIC remote-as 64666 ebgp-multihop 255 update-source Loopback10 address-family ipv4 unicast maximum-prefix 100 90 route-policy blackhole_ipv4_in_public in route-policy drop out long-lived-graceful-restart stale-time send 86400 accept 86400 ! address-family ipv6 unicast maximum-prefix 100 90 route-policy blackhole_ipv6_in_public in route-policy drop out long-lived-graceful-restart stale-time send 86400 accept 86400 ! ! vrf public neighbor 192.0.2.1 use neighbor-group BLACKHOLE_IPV4_PUBLIC description akvorado-1
ForwardingStatus >= 128 as a filter.
While this method is compatible with all routers, it makes the attack successful
as the target is completely unreachable. If your router supports it, Flowspec
can selectively filter flows to stop the attack without impacting the
customer.
flow4 table flowtab4; flow6 table flowtab6; protocol bgp exporter flow4 import none; export where proto = "flowspec4"; ; flow6 import none; export where proto = "flowspec6"; ; # [ ] protocol static flowspec4 flow4; route flow4 dst 203.0.113.68/32; sport = 53; length >= 1476 && <= 1500; proto = 17; bgp_ext_community.add((generic, 0x80060000, 0x00000000)); ; route flow4 dst 203.0.113.206/32; sport = 123; length = 468; proto = 17; bgp_ext_community.add((generic, 0x80060000, 0x00000000)); ; protocol static flowspec6 flow6;
vrf public address-family ipv4 flowspec address-family ipv6 flowspec ! router bgp 12322 address-family vpnv4 flowspec address-family vpnv6 flowspec neighbor-group FLOWSPEC_IPV4_PUBLIC remote-as 64666 ebgp-multihop 255 update-source Loopback10 address-family ipv4 flowspec long-lived-graceful-restart stale-time send 86400 accept 86400 route-policy accept in route-policy drop out maximum-prefix 100 90 validation disable ! address-family ipv6 flowspec long-lived-graceful-restart stale-time send 86400 accept 86400 route-policy accept in route-policy drop out maximum-prefix 100 90 validation disable ! ! vrf public address-family ipv4 flowspec address-family ipv6 flowspec neighbor 192.0.2.1 use neighbor-group FLOWSPEC_IPV4_PUBLIC description akvorado-1
flowspec vrf public address-family ipv4 local-install interface-all ! address-family ipv6 local-install interface-all ! ! !
ForwardingStatus >=
128.
route flow4 dst 203.0.113.68/32; sport = 53; length >= 1476 && <= 1500; proto = 17; bgp_ext_community.add((generic, 0x80060000, 0x00000000)); ;
quantiles(0.1,
0.9)(Bytes/Packets).
The last issue we need to tackle is how to optimize the request: it may need
several seconds to collect the data and it is likely to consume substantial
resources from your ClickHouse database. One solution is to create a
materialized view to pre-aggregate results:
CREATE TABLE ddos_logs ( TimeReceived DateTime, DstAddr IPv6, Proto UInt32, SrcPort UInt16, Gbps SimpleAggregateFunction(sum, Float64), Mpps SimpleAggregateFunction(sum, Float64), sources AggregateFunction(uniqCombined(12), IPv6), countries AggregateFunction(uniqCombined(12), FixedString(2)), size AggregateFunction(quantiles(0.1, 0.9), UInt64) ) ENGINE = SummingMergeTree PARTITION BY toStartOfHour(TimeReceived) ORDER BY (TimeReceived, DstAddr, Proto, SrcPort) TTL toStartOfHour(TimeReceived) + INTERVAL 6 HOUR DELETE ; CREATE MATERIALIZED VIEW ddos_logs_view TO ddos_logs AS SELECT toStartOfMinute(TimeReceived) AS TimeReceived, DstAddr, Proto, SrcPort, sum(((((Bytes * SamplingRate) * 8) / 1000) / 1000) / 1000) / 60 AS Gbps, sum(((Packets * SamplingRate) / 1000) / 1000) / 60 AS Mpps, uniqCombinedState(12)(SrcAddr) AS sources, uniqCombinedState(12)(SrcCountry) AS countries, quantilesState(0.1, 0.9)(toUInt64(Bytes/Packets)) AS size FROM flows WHERE DstNetRole = 'customers' GROUP BY TimeReceived, DstAddr, Proto, SrcPort
ddos_logs table is using the SummingMergeTree engine. When the table
receives new data, ClickHouse replaces all the rows with the same sorting key,
as defined by the ORDER BY directive, with one row which contains summarized
values using either the sum() function or the explicitly specified aggregate
function (uniqCombined and quantiles in our example).3
Finally, we can modify our initial query with the following one:
SELECT * FROM ( SELECT TimeReceived, DstAddr, dictGetOrDefault('protocols', 'name', Proto, '???') AS Proto, SrcPort, sum(Gbps) AS Gbps, sum(Mpps) AS Mpps, uniqCombinedMerge(12)(sources) AS sources, uniqCombinedMerge(12)(countries) AS countries, quantilesMerge(0.1, 0.9)(size) AS size FROM ddos_logs WHERE TimeReceived > now() - INTERVAL 60 MINUTE GROUP BY TimeReceived, DstAddr, Proto, SrcPort ) WHERE (Gbps > 1) OR ((Proto = 'UDP') AND (Gbps > 0.2)) OR ((sources > 20) AND (Gbps > 0.1)) OR ((countries > 10) AND (Gbps > 0.1)) ORDER BY TimeReceived DESC, Gbps DESC
SummingMergeTree tables,import socket import types from clickhouse_driver import Client as CHClient # Put your SQL query here! SQL_QUERY = " " # How many anti-DDoS rules we want at the same time? MAX_DDOS_RULES = 20 def empty_ruleset(): ruleset = types.SimpleNamespace() ruleset.flowspec = types.SimpleNamespace() ruleset.blackhole = types.SimpleNamespace() ruleset.flowspec.v4 = [] ruleset.flowspec.v6 = [] ruleset.blackhole.v4 = [] ruleset.blackhole.v6 = [] return ruleset current_ruleset = empty_ruleset() client = CHClient(host="clickhouse.akvorado.net") while True: results = client.execute(SQL_QUERY) seen = new_ruleset = empty_ruleset() for (t, addr, proto, port, gbps, mpps, sources, countries, size) in results: if (addr, proto, port) in seen: continue seen[(addr, proto, port)] = True # Flowspec if addr.ipv4_mapped: address = addr.ipv4_mapped rules = new_ruleset.flowspec.v4 table = "flow4" mask = 32 nh = "proto" else: address = addr rules = new_ruleset.flowspec.v6 table = "flow6" mask = 128 nh = "next header" if size[0] == size[1]: length = f"length = int(size[0]) " else: length = f"length >= int(size[0]) && <= int(size[1]) " header = f""" # Time: t # Source: address , protocol: proto , port: port # Gbps/Mpps: gbps:.3 / mpps:.3 , packet size: int(size[0]) <=X<= int(size[1]) # Flows: flows , sources: sources , countries: countries """ rules.append( f""" header route table dst address / mask ; sport = port ; length ; nh = socket.getprotobyname(proto) ; bgp_ext_community.add((generic, 0x80060000, 0x00000000)); ; """ ) # Blackhole if addr.ipv4_mapped: rules = new_ruleset.blackhole.v4 else: rules = new_ruleset.blackhole.v6 rules.append( f""" header route address / mask blackhole bgp_community.add((65535, 666)); ; """ ) new_ruleset.flowspec.v4 = list( set(new_ruleset.flowspec.v4[:MAX_DDOS_RULES]) ) new_ruleset.flowspec.v6 = list( set(new_ruleset.flowspec.v6[:MAX_DDOS_RULES]) ) # TODO: advertise changes by mail, chat, ... current_ruleset = new_ruleset changes = False for rules, path in ( (current_ruleset.flowspec.v4, "v4-flowspec"), (current_ruleset.flowspec.v6, "v6-flowspec"), (current_ruleset.blackhole.v4, "v4-blackhole"), (current_ruleset.blackhole.v6, "v6-blackhole"), ): path = os.path.join("/etc/bird/", f" path .conf") with open(f" path .tmp", "w") as f: for r in rules: f.write(r) changes = ( changes or not os.path.exists(path) or not samefile(path, f" path .tmp") ) os.rename(f" path .tmp", path) if not changes: continue proc = subprocess.Popen( ["birdc", "configure"], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) stdout, stderr = proc.communicate(None) stdout = stdout.decode("utf-8", "replace") stderr = stderr.decode("utf-8", "replace") if proc.returncode != 0: logger.error( " error:\n \n ".format( "birdc reconfigure", "\n".join( [" O: ".format(line) for line in stdout.rstrip().split("\n")] ), "\n".join( [" E: ".format(line) for line in stderr.rstrip().split("\n")] ), ) )
FORMAT Markdown to the query.
parser.peg. Here is a simplified rule:
ConditionIPExpr "condition on IP" column:("ExporterAddress"i return "ExporterAddress", nil / "SrcAddr"i return "SrcAddr", nil / "DstAddr"i return "DstAddr", nil ) _ operator:("=" / "!=") _ ip:IP return fmt.Sprintf("%s %s IPv6StringToNum(%s)", toString(column), toString(operator), quote(ip)), nil
ConditionIPExpr. It case-insensitively matches
ExporterAddress, SrcAddr, or DstAddr. The action for each case returns the
proper case for the column name. That s what is stored in the column variable.
Then, it matches one of the possible operators. As there is no code block, it
stores the matched string directly in the operator variable. Then, it tries to
match the IP rule, which is defined elsewhere in the grammar. If it succeeds,
it stores the result of the match in the ip variable and executes the final
action. The action turns the column, operator, and IP into a proper expression
for ClickHouse. For example, if we have ExporterAddress = 203.0.113.15, we
get ExporterAddress = IPv6StringToNum('203.0.113.15').
The IP rule uses a rudimentary regular expression but checks if the matched
address is correct in the action block, thanks to netip.ParseAddr():
IP "IP address" [0-9A-Fa-f:.]+ ip, err := netip.ParseAddr(string(c.text)) if err != nil return "", errors.New("expecting an IP address") return ip.String(), nil
WHERE clause accepted by
ClickHouse:3
WHERE InIfBoundary = 'external' AND ExporterRegion = 'france' AND InIfConnectivity = 'transit' AND SrcAS = 15169 AND DstAddr BETWEEN toIPv6('2a01:e0f:ffff::') AND toIPv6('2a01:e0f:ffff:ffff:ffff:ffff:ffff:ffff')
InputFilter, uses CodeMirror as its foundation and
leverages features such as syntax highlighting, linting, and completion. The
source code for these capabilities can be found in the
codemirror/lang-filter/ directory.
@top Filter expression expression Not expression "(" expression ")" "(" expression ")" And expression "(" expression ")" Or expression comparisonExpression And expression comparisonExpression Or expression comparisonExpression comparisonExpression Column Operator Value Value String Literal ValueLParen ListOfValues ValueRParen ListOfValues ListOfValues ValueComma (String Literal) String Literal // [ ] @tokens // [ ] Column std.asciiLetter (std.asciiLetter std.digit)* Operator $[a-zA-Z!=><]+ String '"' (![\\\n"] "\\" _)* '"'? "'" (![\\\n'] "\\" _)* "'"? Literal (std.digit std.asciiLetter $[.:/])+ // [ ]
SrcAS = 12322 AND (DstAS = 1299 OR SrcAS = 29447) is parsed to:
Filter(Column, Operator, Value(Literal), And, Column, Operator, Value(Literal), Or, Column, Operator, Value(Literal))
export const FilterLanguage = LRLanguage.define( parser: parser.configure( props: [ styleTags( Column: t.propertyName, String: t.string, Literal: t.literal, LineComment: t.lineComment, BlockComment: t.blockComment, Or: t.logicOperator, And: t.logicOperator, Not: t.logicOperator, Operator: t.compareOperator, "( )": t.paren, ), ], ), );
/api/v0/console/filter/validate endpoint accepts a filter and returns a JSON
structure with the errors that were found:
"message": "at line 1, position 12: string literal not terminated", "errors": [ "line": 1, "column": 12, "offset": 11, "message": "string literal not terminated", ]
/api/v0/console/filter/complete endpoint.
Walking the syntax tree was not as easy as I thought, but unit tests helped
a lot.
The backend uses the parser generated by pigeon to complete a column name
or a comparison operator. For values, the completions are either static or
extracted from the ClickHouse database. A user can complete an AS number from
an organization name thanks to the following snippet:
results := []struct Label string ch:"label" Detail string ch:"detail" columnName := "DstAS" sqlQuery := fmt.Sprintf( SELECT concat('AS', toString(%s)) AS label, dictGet('asns', 'name', %s) AS detail FROM flows WHERE TimeReceived > date_sub(minute, 1, now()) AND detail != '' AND positionCaseInsensitive(detail, $1) >= 1 GROUP BY label, detail ORDER BY COUNT(*) DESC LIMIT 20 , columnName, columnName) if err := conn.Select(ctx, &results, sqlQuery, input.Prefix); err != nil c.r.Err(err).Msg("unable to query database") break for _, result := range results completions = append(completions, filterCompletion Label: result.Label, Detail: result.Detail, Quoted: false, )
.proto). The protoc compiler and a language-specific plugin convert it into
code:
$ head flow-4.proto syntax = "proto3"; package decoder; option go_package = "akvorado/inlet/flow/decoder"; message FlowMessagev4 uint64 TimeReceived = 2; uint32 SequenceNum = 3; uint64 SamplingRate = 4; uint32 FlowDirection = 5; $ protoc -I=. --plugin=protoc-gen-go --go_out=module=akvorado:. flow-4.proto $ head inlet/flow/decoder/flow-4.pb.go // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.28.0 // protoc v3.21.12 // source: inlet/flow/data/schemas/flow-4.proto package decoder import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect"
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
initial.txt final.txt
sec/op sec/op vs base
Netflow/with_encoding-12 12.963 2% 7.836 1% -39.55% (p=0.000 n=10)
Sflow/with_encoding-12 19.37 1% 10.15 2% -47.63% (p=0.000 n=10)
Decode() method is a thin layer above
GoFlow2 producer and stores the decoded data into the in-memory structure
generated by protoc. Later, some of the data will be encoded directly during
flow decoding. This is why we measure both the decoding and the
encoding.2
func BenchmarkDecodeEncodeSflow(b *testing.B) r := reporter.NewMock(b) sdecoder := sflow.New(r) data := helpers.ReadPcapPayload(b, filepath.Join("decoder", "sflow", "testdata", "data-1140.pcap")) for _, withEncoding := range []bool true, false title := map[bool]string true: "with encoding", false: "without encoding", [withEncoding] var got []*decoder.FlowMessage b.Run(title, func(b *testing.B) for i := 0; i < b.N; i++ got = sdecoder.Decode(decoder.RawFlow Payload: data, Source: net.ParseIP("127.0.0.1"), ) if withEncoding for _, flow := range got buf := []byte buf = protowire.AppendVarint(buf, uint64(proto.Size(flow))) proto.MarshalOptions .MarshalAppend(buf, flow) )
google.golang.org/protobuf is not the most
efficient one. For a long time, people were relying on gogoprotobuf.
However, the project is now deprecated. A good replacement is
vtprotobuf.3
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
initial.txt bench-2.txt
sec/op sec/op vs base
Netflow/with_encoding-12 12.96 2% 10.28 2% -20.67% (p=0.000 n=10)
Netflow/without_encoding-12 8.935 2% 8.975 2% ~ (p=0.143 n=10)
Sflow/with_encoding-12 19.37 1% 16.67 2% -13.93% (p=0.000 n=10)
Sflow/without_encoding-12 14.62 3% 14.87 1% +1.66% (p=0.007 n=10)
.proto file. The wire format is simple and rely a lot on variable-width
integers.
Variable-width integers, or varints, are an efficient way of encoding unsigned
integers using a variable number of bytes, from one to ten, with small values
using fewer bytes. They work by splitting integers into 7-bit payloads and using
the 8th bit as a continuation indicator, set to 1 for all payloads
except the last.
protowire.AppendTag() encodes a tag,protowire.AppendVarint() encodes a variable-width integer, andprotowire.AppendBytes() append bytes as is.ProtobufIndex) and to generate a proto definition file (fields starting with
Protobuf):
type Column struct Key ColumnKey Name string Disabled bool // [ ] // For protobuf. ProtobufIndex protowire.Number ProtobufType protoreflect.Kind // Uint64Kind, Uint32Kind, ProtobufEnum map[int]string ProtobufEnumName string ProtobufRepeated bool
protowire functions to directly
encode the fields while decoding the flows. They skip disabled fields or
non-repeated fields already encoded. Here is an excerpt of the sFlow
decoder:
sch.ProtobufAppendVarint(bf, schema.ColumnBytes, uint64(recordData.Base.Length)) sch.ProtobufAppendVarint(bf, schema.ColumnProto, uint64(recordData.Base.Protocol)) sch.ProtobufAppendVarint(bf, schema.ColumnSrcPort, uint64(recordData.Base.SrcPort)) sch.ProtobufAppendVarint(bf, schema.ColumnDstPort, uint64(recordData.Base.DstPort)) sch.ProtobufAppendVarint(bf, schema.ColumnEType, helpers.ETypeIPv4)
type FlowMessage struct TimeReceived uint64 SamplingRate uint32 // For exporter classifier ExporterAddress netip.Addr // For interface classifier InIf uint32 OutIf uint32 // For geolocation or BMP SrcAddr netip.Addr DstAddr netip.Addr NextHop netip.Addr // Core component may override them SrcAS uint32 DstAS uint32 GotASPath bool // protobuf is the protobuf representation for the information not contained above. protobuf []byte protobufSet bitset.BitSet
protobuf slice holds encoded data. It is initialized with a capacity of
500 bytes to avoid resizing during encoding. There is also some reserved room at
the beginning to be able to encode the total size as a variable-width integer.
Upon finalizing encoding, the remaining fields are added and the message length
is prefixed:
func (schema *Schema) ProtobufMarshal(bf *FlowMessage) []byte schema.ProtobufAppendVarint(bf, ColumnTimeReceived, bf.TimeReceived) schema.ProtobufAppendVarint(bf, ColumnSamplingRate, uint64(bf.SamplingRate)) schema.ProtobufAppendIP(bf, ColumnExporterAddress, bf.ExporterAddress) schema.ProtobufAppendVarint(bf, ColumnSrcAS, uint64(bf.SrcAS)) schema.ProtobufAppendVarint(bf, ColumnDstAS, uint64(bf.DstAS)) schema.ProtobufAppendIP(bf, ColumnSrcAddr, bf.SrcAddr) schema.ProtobufAppendIP(bf, ColumnDstAddr, bf.DstAddr) // Add length and move it as a prefix end := len(bf.protobuf) payloadLen := end - maxSizeVarint bf.protobuf = protowire.AppendVarint(bf.protobuf, uint64(payloadLen)) sizeLen := len(bf.protobuf) - end result := bf.protobuf[maxSizeVarint-sizeLen : end] copy(result, bf.protobuf[end:end+sizeLen]) return result
-benchmem flag to monitor allocation
numbers. Each allocation incurs an additional cost to the garbage collector. The
Go profiler is a valuable tool for identifying areas of code that can be
optimized:
$ go test -run=__nothing__ -bench=Netflow/with_encoding \ > -benchmem -cpuprofile profile.out \ > akvorado/inlet/flow goos: linux goarch: amd64 pkg: akvorado/inlet/flow cpu: AMD Ryzen 5 5600X 6-Core Processor Netflow/with_encoding-12 143953 7955 ns/op 8256 B/op 134 allocs/op PASS ok akvorado/inlet/flow 1.418s $ go tool pprof profile.out File: flow.test Type: cpu Time: Feb 4, 2023 at 8:12pm (CET) Duration: 1.41s, Total samples = 2.08s (147.96%) Entering interactive mode (type "help" for commands, "o" for options) (pprof) web
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
bench-2.txt bench-3.txt
sec/op sec/op vs base
Netflow/with_encoding-12 10.284 2% 7.758 3% -24.56% (p=0.000 n=10)
Netflow/without_encoding-12 8.975 2% 7.304 2% -18.61% (p=0.000 n=10)
Sflow/with_encoding-12 16.67 2% 14.26 1% -14.50% (p=0.000 n=10)
Sflow/without_encoding-12 14.87 1% 13.56 2% -8.80% (p=0.000 n=10)
github.com/jhump/protoreflect: the
protoparse package parses the proto definition file we generate and the
dynamic package decodes the messages. Check the ProtobufDecode()
method for more details.4
To get the final figures, I have also optimized the decoding in GoFlow2. It
was relying heavily on binary.Read(). This function may use
reflection in certain cases and each call allocates a byte array to read data.
Replacing it with a more efficient version provides the following
improvement:
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
bench-3.txt bench-4.txt
sec/op sec/op vs base
Netflow/with_encoding-12 7.758 3% 7.365 2% -5.07% (p=0.000 n=10)
Netflow/without_encoding-12 7.304 2% 6.931 3% -5.11% (p=0.000 n=10)
Sflow/with_encoding-12 14.256 1% 9.834 2% -31.02% (p=0.000 n=10)
Sflow/without_encoding-12 13.559 2% 9.353 2% -31.02% (p=0.000 n=10)
Notice Some paragraphs were editorialized by ChatGPT, using editorialize and keep it short as a prompt. The result was proofread by a human for correctness. The main idea is that ChatGPT should be better at English than me.
vtprotobuf generates more optimized Go code by removing an
abstraction layer. It directly generates the code encoding each field to
bytes:
if m.OutIfSpeed != 0 i = encodeVarint(dAtA, i, uint64(m.OutIfSpeed)) i-- dAtA[i] = 0x6 i-- dAtA[i] = 0xd8
protoprint package to generate proto definition
file. I did not use it.
variable "hcloud_token" sensitive = true provider "hcloud" token = var.hcloud_token resource "hcloud_server" "web03" name = "web03" server_type = "cpx11" image = "debian-11" datacenter = "nbg1-dc3" resource "hcloud_rdns" "rdns4-web03" server_id = hcloud_server.web03.id ip_address = hcloud_server.web03.ipv4_address dns_ptr = "web03.luffy.cx" resource "hcloud_rdns" "rdns6-web03" server_id = hcloud_server.web03.id ip_address = hcloud_server.web03.ipv6_address dns_ptr = "web03.luffy.cx"
import App, TerraformStack, Fn from "cdktf"; import HcloudProvider from "./.gen/providers/hcloud/provider"; import * as hcloud from "./.gen/providers/hcloud"; class MyStack extends TerraformStack constructor(scope: Construct, name: string) super(scope, name); const hcloudToken = new TerraformVariable(this, "hcloudToken", type: "string", sensitive: true, ); const hcloudProvider = new HcloudProvider(this, "hcloud", token: hcloudToken.value, ); const web03 = new hcloud.server.Server(this, "web03", name: "web03", serverType: "cpx11", image: "debian-11", datacenter: "nbg1-dc3", provider: hcloudProvider, ); new hcloud.rdns.Rdns(this, "rdns4-web03", serverId: Fn.tonumber(web03.id), ipAddress: web03.ipv4Address, dnsPtr: "web03.luffy.cx", provider: hcloudProvider, ); new hcloud.rdns.Rdns(this, "rdns6-web03", serverId: Fn.tonumber(web03.id), ipAddress: web03.ipv6Address, dnsPtr: "web03.luffy.cx", provider: hcloudProvider, ); const app = new App(); new MyStack(app, "cdktf-take1"); app.synth();
cdktf synth generates a configuration file for Terraform, terraform
plan previews the changes, and terraform apply applies them. Now that you
have a general-purpose language, you can use functions.
bernat.ch zone with this snippet of code:
D("bernat.ch", REG_NONE, DnsProvider(DNS_BIND, 0), DnsProvider(DNS_GANDI), DefaultTTL('2h'), FastMailMX('bernat.ch', subdomains: ['vincent'] ), WebServers('@'), WebServers('vincent');
new Route53Zone(this, "bernat.ch", providers.aws) .sign(dnsCMK) .registrar(providers.gandiVB) .www("@", servers) .www("vincent", servers) .www("media", servers) .fastmailMX(["vincent"]);
Route53Zone() creates a new zone hosted by Route 53,sign() signs the zone with the provided master key,registrar() registers the zone to the registrar of the domain and sets up DNSSEC,www() creates A and AAAA records for the provided name pointing to the web servers,fastmailMX() creates the MX records and other support records to direct emails to Fastmail.fastmailMX() function. It generates a few records
and returns the current zone for chaining:
fastmailMX(subdomains?: string[]) (subdomains ?? []) .concat(["@", "*"]) .forEach((subdomain) => this.MX(subdomain, [ "10 in1-smtp.messagingengine.com.", "20 in2-smtp.messagingengine.com.", ]) ); this.TXT("@", "v=spf1 include:spf.messagingengine.com ~all"); ["mesmtp", "fm1", "fm2", "fm3"].forEach((dk) => this.CNAME( $ dk ._domainkey , $ dk .$ this.name .dkim.fmhosted.com. ) ); this.TXT("_dmarc", "v=DMARC1; p=none; sp=none"); return this;
Makefile automatically installs Pulumi using a curl sh
pattern and does not work with /bin/sh. There is a lack of
interest for community-based contributions4 or even for providers for
smaller players.
environment.systemPackages = with pkgs; [ bat htop liboping mg mtr ncdu tmux ];
services.nginx = enable = true; package = (pkgs.nginxStable.override withStream = false; modules = with pkgs.nginxModules; [ brotli ipscrub ]; openssl = pkgs.openssl_1_1; );
services.nginx.package = pkgs.nginxStable.overrideAttrs (old: patches = oldAttrs.patches ++ [ # HTTP/2: reject zero length headers with PROTOCOL_ERROR. (pkgs.fetchpatch url = https://github.com/nginx/nginx/commit/dbdd[ ].patch; sha256 = "a48190[ ]"; ) # HTTP/2: limited number of DATA frames. (pkgs.fetchpatch url = https://github.com/nginx/nginx/commit/94c5[ ].patch; sha256 = "af591a[ ]"; ) # HTTP/2: limited number of PRIORITY frames. (pkgs.fetchpatch url = https://github.com/nginx/nginx/commit/39bb[ ].patch; sha256 = "1ad8fe[ ]"; ) ]; );
common.nix contains the configuration to be applied to any host
(SSH, users, common software packages), web.nix contains the
configuration for the web servers, isso.nix runs Isso into a
systemd container.
/etc/nixos/configuration.nix
file. After modifying it, you have to run nixos-rebuild switch. Nix fetches
all possible dependencies from the binary cache and builds the remaining
packages. It creates a new entry in the boot loader menu and activates the new
configuration.
To manage several nodes, there exists several options, including NixOps,
deploy-rs, Colmena, and morph. I do not know all of them, but from
my point of view, the differences are not that important. It is also possible to
build such a tool yourself as Nix provides the most important building blocks:
nix build and nix copy. NixOps is one of the first tools available but I
encourage you to explore the alternatives.
NixOps configuration is written in Nix. Here is a simplified configuration
to deploy znc01.luffy.cx, web01.luffy.cx, and web02.luffy.cx, with the
help of the server and web functions:
let server = hardware: name: imports: deployment.targetHost = "$ name .luffy.cx"; networking.hostName = name; networking.domain = "luffy.cx"; imports = [ (./hardware/. + "/$ hardware .nix") ] ++ imports; ; web = hardware: idx: imports: server hardware "web$ lib.fixedWidthNumber 2 idx " ([ ./web.nix ] ++ imports); in network.description = "Luffy infrastructure"; network.enableRollback = true; defaults = import ./common.nix; znc01 = server "exoscale" [ ./znc.nix ]; web01 = web "hetzner" 1 [ ./isso.nix ]; web02 = web "hetzner" 2 [];
flake.nix and
flake.lock at the root of some repositories.
As a short example, here is the flake.nix content shipped with Snimpy, an
interactive SNMP tool for Python relying on libsmi, a C library:
inputs = nixpkgs.url = "nixpkgs"; flake-utils.url = "github:numtide/flake-utils"; ; outputs = self, ... @inputs: inputs.flake-utils.lib.eachDefaultSystem (system: let pkgs = inputs.nixpkgs.legacyPackages."$ system "; in # nix build packages.default = pkgs.python3Packages.buildPythonPackage name = "snimpy"; src = self; preConfigure = ''echo "1.0.0-0-000000000000" > version.txt''; checkPhase = "pytest"; checkInputs = with pkgs.python3Packages; [ pytest mock coverage ]; propagatedBuildInputs = with pkgs.python3Packages; [ cffi pysnmp ipython ]; buildInputs = [ pkgs.libsmi ]; ; # nix run + nix shell apps.default = type = "app"; program = "$ self.packages."$ system ".default /bin/snimpy"; ; # nix develop devShells.default = pkgs.mkShell name = "snimpy-dev"; buildInputs = [ self.packages."$ system ".default.inputDerivation pkgs.python3Packages.ipython ]; ; );
nix run github:vincentbernat/snimpy runs Snimpy,nix shell github:vincentbernat/snimpy provides a shell with Snimpy ready-to-use,nix build github:vincentbernat/snimpy builds the Python package, tests included, andnix develop . provides a shell to hack around Snimpy when run from a fresh checkout.9flake.nix file to set up a shell with Terraform and
CDKTF installed and with the appropriate environment variables to automate my
infrastructure.
Terraform is already packaged in nixpkgs. However, I need to apply a patch
on top of the Gandi provider. Not a problem with Nix!
terraform = pkgs.terraform.withPlugins (p: [ p.aws p.hcloud p.vultr (p.gandi.overrideAttrs (old: src = pkgs.fetchFromGitHub owner = "vincentbernat"; repo = "terraform-provider-gandi"; rev = "feature/livedns-key"; hash = "sha256-V16BIjo5/rloQ1xTQrdd0snoq1OPuDh3fQNW7kiv/kQ="; ; )) ]);
package.json file with all the dependencies
needed, including the ones to use TypeScript as the language to define
infrastructure:
"name": "cdktf-take1", "version": "1.0.0", "main": "main.js", "types": "main.ts", "private": true, "dependencies": "@types/node": "^14.18.30", "cdktf": "^0.13.3", "cdktf-cli": "^0.13.3", "constructs": "^10.1.151", "eslint": "^8.27.0", "prettier": "^2.7.1", "ts-node": "^10.9.1", "typescript": "^3.9.10", "typescript-language-server": "^2.1.0"
yarn.lock file that can be
used directly to declare a derivation containing all the dependencies:
nodeEnv = pkgs.mkYarnModules pname = "cdktf-take1-js-modules"; version = "1.0.0"; packageJSON = ./package.json; yarnLock = ./yarn.lock; ;
cdktfProviders = pkgs.stdenvNoCC.mkDerivation name = "cdktf-providers"; nativeBuildInputs = [ pkgs.nodejs terraform ]; src = nix-filter root = ./.; include = [ ./cdktf.json ./tsconfig.json ]; ; buildPhase = '' export HOME=$(mktemp -d) export CHECKPOINT_DISABLE=1 export DISABLE_VERSION_CHECK=1 export PATH=$ nodeEnv /node_modules/.bin:$PATH ln -nsf $ nodeEnv /node_modules node_modules # Build all providers we have in terraform for provider in $(cd $ terraform /libexec/terraform-providers; echo */*/*/*); do version=''$ provider##*/ provider=''$ provider%/* echo "Build $provider@$version" cdktf provider add --force-local $provider@$version cat done echo "Compile TS JS" tsc ''; installPhase = '' mv .gen $out ln -nsf $ nodeEnv /node_modules $out/node_modules ''; ;
devShells.default = pkgs.mkShell name = "cdktf-take1"; buildInputs = [ pkgs.nodejs pkgs.yarn terraform ]; shellHook = '' # No telemetry export CHECKPOINT_DISABLE=1 # No autoinstall of plugins export CDKTF_DISABLE_PLUGIN_CACHE_ENV=1 # Do not check version export DISABLE_VERSION_CHECK=1 # Access to node modules export PATH=$PWD/node_modules/.bin:$PATH ln -nsf $ nodeEnv /node_modules node_modules ln -nsf $ cdktfProviders .gen # Credentials for p in \ njf.nznmba.pbz/Nqzvavfgengbe \ urgmare.pbz/ivaprag@oreang.pu \ ihyge.pbz/ihyge@ivaprag.oreang.pu; do eval $(pass show $(echo $p tr 'A-Za-z' 'N-ZA-Mn-za-m') grep '^export') done eval $(pass show personal/cdktf/secrets grep '^export') export TF_VAR_hcloudToken="$HCLOUD_TOKEN" export TF_VAR_vultrApiKey="$VULTR_API_KEY" unset VULTR_API_KEY HCLOUD_TOKEN ''; ;
buildInputs are available in the provided shell.
The content of shellHook is sourced when starting the shell. It sets up some
symbolic links to make the JavaScript environment built at an earlier step
available, as well as the generated CDKTF providers. It also exports all the
credentials.10
I am also using direnv with an .envrc to
automatically load the development environment. This also enables the
environment to be available from inside Emacs, notably when using lsp-mode
to get TypeScript completions. Without direnv, nix develop . can activate
the environment.
I use the following commands to deploy the infrastructure:11
$ cdktf synth $ cd cdktf.out/stacks/cdktf-take1 $ terraform plan --out plan $ terraform apply plan $ terraform output -json > ~-automation/nixops-take1/cdktf.json
"hardware": "hetzner", "ipv4Address": "5.161.44.145", "ipv6Address": "2a01:4ff:f0:b91::1", "name": "web05.luffy.cx", "tags": [ "web", "continent:NA", "continent:SA" ]
network.nix, this list is imported and
transformed into an attribute set describing the servers. A simplified version
looks like this:
let lib = inputs.nixpkgs.lib; shortName = name: builtins.elemAt (lib.splitString "." name) 0; domainName = name: lib.concatStringsSep "." (builtins.tail (lib.splitString "." name)); server = hardware: name: imports: networking = hostName = shortName name; domain = domainName name; ; deployment.targetHost = name; imports = [ (./hardware/. + "/$ hardware .nix") ] ++ imports; ; cdktf-servers-json = (lib.importJSON ./cdktf.json).servers.value; cdktf-servers = map (s: let tags-maybe-import = map (t: ./. + "/$ t .nix") s.tags; tags-import = builtins.filter (t: builtins.pathExists t) tags-maybe-import; in name = shortName s.name; value = server s.hardware s.name tags-import; ) cdktf-servers-json; in // [ ] // builtins.listToAttrs cdktf-servers
web05, this expands to:
web05 = networking = hostName = "web05"; domainName = "luffy.cx"; ; deployment.targetHost = "web05.luffy.cx"; imports = [ ./hardware/hetzner.nix ./web.nix ]; ;
flake.nix file to set up a shell with
NixOps configured. Because NixOps do not support rollouts, I usually use the
following commands to deploy on a single server:12
$ nix flake update $ nixops deploy --include=web04 $ ./tests web04.luffy.cx
$ (set -e; for h in web 03..06 ; do nixops deploy --include=$h; done)
nixops deploy rolls out all servers in parallel and therefore could cause a
short outage where all Nginx are down at the same time.
apply()
method on an output. It makes it easy to transform data that are not known
during the planning stage. Terraform has functions to
serve a similar purpose, but they are more limited.
/bin/bash. I also submitted PR #56,
which was merged 4 months later and quickly reverted without
an explanation.
. with github:vincentbernat/snimpy,
like in the other commands, but having Snimpy dependencies without
Snimpy source code is less interesting.
cdktf command can wrap the terraform commands, but I
prefer to use them directly as they are more flexible.
TL;DR
Never trust show commit changes diff on Cisco IOS XR.
commit command to apply it to the running
configuration.1 This is a common concept for many NOS.
Before committing the candidate configuration to the running configuration, you
may want to check the changes that have accumulated until now. That s where the
show commit changes diff command2 comes up. Its goal is to show the
difference between the running configuration (show running-configuration) and
the candidate configuration (show configuration merge). How hard can it be?
Let s put an interface down on IOS XR 7.6.2 (released in August 2022):
RP/0/RP0/CPU0:router(config)#int Hu0/1/0/1 shut RP/0/RP0/CPU0:router(config)#show commit changes diff Wed Nov 23 11:08:30.275 CET Building configuration... !! IOS XR Configuration 7.6.2 + interface HundredGigE0/1/0/1 + shutdown ! end
+ sign before interface HundredGigE0/1/0/1 makes it look like you did
create a new interface. Maybe there was a typo? No, the diff is just broken. If
you look at the candidate configuration, everything is like you expect:
RP/0/RP0/CPU0:router(config)#show configuration merge int Hu0/1/0/1 Wed Nov 23 11:08:43.360 CET interface HundredGigE0/1/0/1 description PNI: (some description) bundle id 4000 mode active lldp receive disable transmit disable ! shutdown load-interval 30
RP/0/RP0/CPU0:router(config)#no int GigabitEthernet 0/0/0/5 RP/0/RP0/CPU0:router(config)#int TenGigE 0/0/0/5 shut RP/0/RP0/CPU0:router(config)#no int TenGigE 0/0/0/28 RP/0/RP0/CPU0:router(config)#int TenGigE 0/0/0/28 shut RP/0/RP0/CPU0:router(config)#no int TenGigE 0/0/0/29 RP/0/RP0/CPU0:router(config)#int TenGigE 0/0/0/29 shut RP/0/RP0/CPU0:router(config)#show commit changes diff Mon Nov 7 15:07:22.990 CET Building configuration... !! IOS XR Configuration 7.2.2 - interface GigabitEthernet0/0/0/5 - shutdown ! + interface TenGigE0/0/0/5 + shutdown ! interface TenGigE0/0/0/28 - description Trunk (some description) - bundle id 2 mode active ! end
GigabitEthernet0/0/0/5 and create TenGigE0/0/0/5. The two
next commands are also correctly represented by the last chunk of the diff.
TenGigE0/0/0/28 was already shut down, so it is expected that only
description and bundle id are removed. However, the diff command forgets
about the modifications for TenGigE0/0/0/29. The diff should include a chunk
similar to the last one.
RP/0/RP0/CPU0:router(config)#show run int TenGigE 0/0/0/29 Mon Nov 7 15:07:43.571 CET interface TenGigE0/0/0/29 description Trunk to other router bundle id 2 mode active shutdown ! RP/0/RP0/CPU0:router(config)#show configuration merge int TenGigE 0/0/0/29 Mon Nov 7 15:07:53.584 CET interface TenGigE0/0/0/29 shutdown !
TenGigE0/0/0/28 but incorrect for
TenGigE0/0/0/29 while they have the same configuration? How can you trust the
diff command if it forgets part of the configuration?
Do you remember the last time you ran an Ansible playbook and discovered the
whole router ospf block disappeared without a warning? If you use automation
tools, you should check how the diff is assembled. Automation tools should build
it from the result of show running-config and show configuration merge. This
is what NAPALM does. This is not what cisco.iosxr
collection for Ansible does.
The problem is not limited to the interface directives. You can get similar
issues for other parts of the configuration. For example, here is what we get
when removing inactive BGP neighbors on IOS XR 7.2.2:
RP/0/RP0/CPU0:router(config)#router bgp 65400 RP/0/RP0/CPU0:router(config-bgp)#vrf public RP/0/RP0/CPU0:router(config-bgp-vrf)#no neighbor 217.29.66.1 RP/0/RP0/CPU0:router(config-bgp-vrf)#no neighbor 217.29.66.75 RP/0/RP0/CPU0:router(config-bgp-vrf)#no neighbor 217.29.66.110 RP/0/RP0/CPU0:router(config-bgp-vrf)#no neighbor 217.29.66.112 RP/0/RP0/CPU0:router(config-bgp-vrf)#no neighbor 217.29.66.158 RP/0/RP0/CPU0:router(config-bgp-vrf)#show commit changes diff Tue Aug 2 13:58:02.536 CEST Building configuration... !! IOS XR Configuration 7.2.2 router bgp 65400 vrf public - neighbor 217.29.66.1 - remote-as 16004 - use neighbor-group MIX_IPV4_PUBLIC - description MIX: MIX-IT ! - neighbor 217.29.66.75 - remote-as 49367 - use neighbor-group MIX_IPV4_PUBLIC ! - neighbor 217.29.67.10 - remote-as 19679 ! - neighbor 217.29.67.15 - neighbor 217.29.66.112 - remote-as 8075 - use neighbor-group MIX_IPV4_PUBLIC - description MIX: Microsoft - address-family ipv4 unicast - maximum-prefix 1500 95 restart 5 ! ! - neighbor 217.29.66.158 - remote-as 24482 - use neighbor-group MIX_IPV4_PUBLIC - description MIX: SG.GS - address-family ipv4 unicast ! ! ! ! end
show run and show configuration merge. The output
would always be correct!
router bgp directive. Such a
limitation is a great pain for both operations and automation.
show commit, as show commit
changes diff is the only valid command you can execute from this point.
Starting from IOS XR 7.5.1, show commit changes diff precise is also a
valid command. However, I have failed to find any documentation about it and
it seems to provide the same output as show commit changes diff. That s
how clunky IOS XR can be.
Azide-1.25mm_001.dat, but of course, the purpose of this tutorial is to enable you to work on all of them. The data files contain the time evolution of the absorbance for all wavelengths, in a matrixformat, in which each row correpond to a time point and each column to a wavelength. Start QSoas, and launch the command:
QSoas> load /comments='"'Then, choose the
Azide-1.25mm_001.dat data file. This should bring up a horizontal red line at the bottom of the data display, with X values between about 0 and 2.5. If you zoom on the red line with the mouse wheel, you'll realize it is data. The /comments='"' part is very important since it allows the extraction of the wavelength from the data. We will look at what it means another day. At this stage, you can look at the loaded data using the command:
QSoas> editYou should have a window looking like this:
nan as the X value and what is obviously the wavelength for all the Y values. To tell that QSoas should take this line as the wavelength (which will be the perpendicular coordinate, the coordinate of the other direction of the matrix), first close the edit window and run:
QSoas> set-perp /from-row=0Splitting and fitting Now, we have a single dataset containing a lot of Y columns. We want to fit all of them simultaneously with a (mono) exponential fit. For that, we first need to split the big matrix into a series of X,Y datasets (because fitting only works on the first Y). This is possible by running:
QSoas> expand /style=red-to-blue /flags=kineticsYour screen should now look like this:
/style=red-to-blue is not strictly necessary, but it gives the red to blue color gradient which makes things easier to look at (and cooler !). The /flags=kinetics is there to attach a label (a flag) to the newly created datasets so we can easily manipulate all of them at the same time. Then it's time to fit, with the following command:
QSoas> mfit-exponential-decay flagged:kineticsThis should bring up a new window. After resizing it, you should have something that looks like this:
Z = 728.15 showing that QSoas has recognized that the currently displayed dataset corresponds to the wavelength 728.15. The equation fitted to the data is: $$y(x) = A_\infty + A_1 \times \exp -(x - x_0)/\tau_1$$
In this case, while the \(A_1\) and \(A_\infty\) parameters clearly depend on the wavelength, the time constant of evolution should be independent of wavelength (the process happens at a certain rate regardless of the wavelength we're analyzing), so that the \(\tau_1\) parameter should be common for all the datasets/wavelengths. Just click on the globalcheckbox at the right of the
tau_1 parameter, make sure it is checked, and hit the Fitbutton... The fit should not take long (less than a minute), and then you end up with the results of the fits: all the parameters. The best way to look at the non global parameters like \(A_1\) and \(A_\infty\) is to use the
Show Parametersitem from the
Parametersmenu. Using it and clicking on
A_inf too should give you a display like this one:
A_inf parameter corresponds to the spectum at infinite time (of azide-bound heme), while the A_1 parameter corresponds to the difference spectrum between the initial (azide-free) and final (azide-bound) states.
Now, the fit is finished, you can save the parameters if you want to reload them in a later fit by using the Parameters/
Savemenu item or export them in a form more suitable for plotting using
Parameters/
Export(although QSoas can also display and the parameters saved using
Save). This concludes this first approach to fitting the data. What you can do is
tau_1 parameter as a function of the azide concentration;QSoas> mfit-exponential-decay /exponentials=2 flagged:kinetics
QSoas> in the code areas above are meant to be typed into the QSoas command line (at the bottom of the window), and started by pressing enter at the end. You must remove the QSoas> bit. The other lines (when applicable) show you the response of QSoas, in the terminal just above the command-line. You may want to play with the QSoas tutorial to learn more about how to interact with QSoas.
About QSoas
QSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 3.1. You can freely (and at no cost) download its source code or precompiled versions for MacOS and Windows there. Alternatively, you can clone from the GitHub repository.
kinetic-system fits, and the handling of Marcus-Hush-Chidsey (or Marcus "distribution of states") kinetics for electron transfers.
Column and row names
This release greatly improves the handling of column and row names, including commands to easily modify them, the possibility to use Ruby formulas to change them, and a much better way read and write them to data files. Mastering the use of column names (and to a lesser extent, row names) can greatly simplify data handling, especially when dealing with files with a large number of columns.
Complex numbers
Version 3.1 brings in support for formulas handling complex numbers. Although it is not possible to store complex numbers directly into datasets, it is easy to separate them in real and imaginary parts to your liking.
Scripting improvement
Two important improvements for scripting are included in version 3.1. The first is the possibility to define virtual files inside a script file, which makes it easy to define subfunctionsto run using commands like run-for-each. The second is the possibility to define
variablesto be reused later (like the script arguments) using the new command let. There are a lot of other new features, improvements and so on, look for the full list there. About QSoas
flows_1m0s aggregated
table, unless I request something about ports and IP addresses.
en-FR language available, which would
have been useful for my 
Update (2022-01) XScreenSaver can also run arbitrary programs as a screen saver.
startx and a systemd service:
[Unit] Description=X11 session for bernat After=graphical.target systemd-user-sessions.service [Service] User=bernat WorkingDirectory=~ PAMName=login Environment=XDG_SESSION_TYPE=x11 TTYPath=/dev/tty8 StandardInput=tty UnsetEnvironment=TERM UtmpIdentifier=tty8 UtmpMode=user StandardOutput=journal ExecStartPre=/usr/bin/chvt 8 ExecStart=/usr/bin/startx -- vt8 -keeptty -verbose 3 -logfile /dev/null Restart=no [Install] WantedBy=graphical.target
systemd-user-sessions.service, which enables
user logins after boot by removing the /run/nologin file.
User=bernat, the unit is started with the identity of the
specified user. This implies that Xorg does not run with elevated
privileges.
PAMName=login, the executed process is registered as a PAM
session for the login service, which includes pam_systemd.
This module registers the session to the systemd login manager.
To be effective, we also need to allocate a TTY with
TTYPath=/dev/tty8. When the TTY is active, the user is granted
additional access to local devices notably display, sound, keyboard,
mouse. These additional rights are needed to get Xorg working
rootless.1 The TERM environment variable is unset because
it would be set to linux by systemd as a result of attaching the
standard input to the TTY. Moreover, we inform pam_systemd we want
an X11 session with Environment=XDG_SESSION_TYPE=x11. Otherwise,
logind considers the session idle unless it receives input on
the TTY. Software relying on the idle hint from logind would be
ineffective.2
UtmpIdentifier=tty8 and UtmpMode=user directives are just
a nice addition to register the session in /var/run/utmp.
Xorg through startx. For logind to
allow Xorg to take control of the local devices, chvt 8 switches
to the allocated TTY.3 StandardOutput=journal, combined
with the -verbose 3 -logfile /dev/null flags for Xorg, puts the
logs from the X server into the journal instead of using a file.
While equal to the default value, the Restart=no directive
highlights we do not want this unit to be restarted. This ensures
the loginless session is only available on boot. By default,
startx runs xinitrc. If you want to run Kodi instead, add
/usr/bin/kodi-standalone between startx and --.
/etc/systemd/system/x11-autologin.service and
enable it with systemctl enable x11-autologin.service. Xorg is now
running rootless and logging into the journal. After using it for a
few months, I didn t notice any regression compared to LightDM with
autologin.
logind provides access to
devices, see this blog post. The method names do not match the
current implementation, but the concepts are still correct. Xorg
takes control of the session when the TTY is active.
xf86OpenConsole: VT_ACTIVATE
failed: Operation not permitted.
NetBox intends to represent the desired state of a network versus its operational state. As such, automated import of live network state is strongly discouraged. All data created in NetBox should first be vetted by a human to ensure its integrity. NetBox can then be used to populate monitoring and provisioning systems with a high degree of confidence.When introducing Jerikan, a common feedback we got was: you should use NetBox for this. Indeed, Jerikan s source of truth is a bunch of YAML files versioned with Git.
Infrastructure as code is an approach to infrastructure automation based on practices from software development. It emphasizes consistent, repeatable routines for provisioning and changing systems and their configuration. You make changes to code, then use automation to test and apply those changes to your systems. Kief Morris, Infrastructure as Code, O Reilly.A version control system is a central tool for infrastructure as code. The usual candidate is Git with a source code management system like GitLab or GitHub. You get:
Update (2021-11) Nautobot, a fork of NetBox, will soon address this point by using Dolt, an SQL database engine allowing you to clone, branch, and merge, like a Git repository. Dolt is compatible with MySQL clients. See Nautobots, Roll Back! for a preview of this feature.
scp -3 can copy files between two remote hosts through localhost.
This comes in handy when the two servers cannot communicate
directly or if they are unable to authenticate one to the
other.1 Unfortunately, rsync does not support such a feature.
Here is a trick to emulate the behavior of scp -3 with SSH tunnels.
When syncing with a remote host, rsync invokes ssh to spawn a
remote rsync --server process. It interacts with it through its
standard input and output. The idea is to recreate the same setup
using SSH tunnels and socat, a versatile tool to establish
bidirectional data transfers.
The first step is to connect to the source server and ask rsync the
command-line to spawn the remote rsync --server process. The -e
flag overrides the command to use to get a remote shell: instead of
ssh, we use echo.
$ ssh web04 $ rsync -e 'sh -c ">&2 echo $@" echo' -aLv /data/. web05:/data/. web05 rsync --server -vlogDtpre.iLsfxCIvu . /data/. rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]
socat. When receiving the connection, socat spawns the
rsync --server command we got at the previous step and connects its
standard input and output to the incoming TCP socket.
$ ssh -L 127.0.0.1:5000:127.0.0.1:5000 web05 $ socat tcp-listen:5000,reuseaddr exec:"rsync --server -vlogDtpre.iLsfxCIvu . /data/."
socat is used in place of a regular SSH connection and
connects its standard input and output to a TCP socket connected to the
remote port 5000. Thanks to the remote port forwarding, SSH forwards
the data to the local port 5000. From there, it is relayed back to the
destination, as described in the previous step.
$ ssh -R 127.0.0.1:5000:127.0.0.1:5000 web04 $ rsync -e 'sh -c "socat stdio tcp-connect:127.0.0.1:5000"' -aLv /data/. remote:/data/. sending incremental file list haproxy.debian.net/ haproxy.debian.net/dists/buster-backports-1.8/Contents-amd64.bz2 haproxy.debian.net/dists/buster-backports-1.8/Contents-i386.bz2 [ ] media.luffy.cx/videos/2021-frnog34-jerikan/progressive.mp4 sent 921,719,453 bytes received 26,939 bytes 7,229,383.47 bytes/sec total size is 7,526,872,300 speedup is 8.17
--server. Use this
hack at your own risk!
The options--serverand--senderare used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these options may be needed in certain scenarios, such as when setting up a login that can only run an rsync command. For instance, the support directory of the rsync distribution has an example script namedrrsync(for restricted rsync) that can be used with a restricted ssh login.
$ socat \ > exec:"ssh web04 rsync --server --sender -vlLogDtpre.iLsfxCIvu . /data/." \ > exec:"ssh web05 rsync --server -vlogDtpre.iLsfxCIvu /data/. /data/." \ over-long vstring received (511 > 255) over-long vstring received (511 > 255) rsync error: requested action not supported (code 4) at compat.c(387) [sender=3.2.3] rsync error: requested action not supported (code 4) at compat.c(387) [Receiver=3.2.3] socat[878291] E waitpid(): child 878292 exited with status 4 socat[878291] E waitpid(): child 878293 exited with status 4
Next.