Sven Hoexter: Google Cloud Oddities Summer 2025 Edition
Latest oddities I ran into with Google Cloud products before I
start to forget about them again.
e2 Compute Instances vs CloudNAT
Years ago I already had a surprising encounter with the Google Cloud
e2 instances. Back then we observed CPU steal time from 20-60%, which
made the instances unusable for anything remotely latency sensitive.
Now someone started to run a workload which creates many outbound
connections to the same destination IP:Port. To connect to the internet
we utilize the Google Cloud product "CloudNAT" which implements a NAT
solution somewhere in the network layer.
Starting the workload let after a few seconds to all sort of connection
issues, and of course logs from CloudNAT that it dropped connections.
The simplest reproducer I could find was
while true; do curl http://sven.stormbind.net; done
which already let to connection drops on CloudNAT.
We starred a bit at output of gcloud compute routers get-nat-mapping-info our-default-natgw,
but allocating additional ports still worked fine in general. Further investigation
let to two differences between a project which was fine and those that failed:
- c2d or n2d machine types instead of e2 and
- usage of gVNIC.
min_ports_per_vm value too low
and it could not allocate new ports in time. Thus we did some additional optimizations:
- raised
min_ports_per_vmto 256 - raised
max_ports_per_vmto 32768 (the sensible maximum because CloudNAT will always double its allocation) - set
nat_tcp_timewait_secto 30, default is 120, reclaim of ports is only running every 30s, thus ports can be re-used after 30-60s
kind: Service with spec.type: LoadBalancer. The
logic documented by Google Cloud
is that the forwarding rule will have per port configuration if it's five or less, and above
that it will open for all ports. Sadly that does not work e.g. in cases where you've an
internal load balancer and a serviceAttachment attached to the forwarding rule. In my
experience reconfiguring was also unreliable in cases without a serviceAttachment and
required a manual deletion of the service load balancer to have the operator reconcile it
and create it correctly.
Given that we wanted to have all ports open to allow us to dynamically add more ports on
a specific load balancer, but there is no annotation for that, I worked around with this
beauty:
ports:
- name: dummy-0
port: 2342
protocol: TCP
targetPort: 2342
- name: dummy-1
port: 2343
protocol: TCP
targetPort: 2343
- name: dummy-2
port: 2344
protocol: TCP
targetPort: 2344
- name: dummy-3
port: 2345
protocol: TCP
targetPort: 2345
- name: service-1
port: 4242
protocol: TCP
targetPort: 4242
- name: service-2
port: 4343
protocol: TCP
targetPort: 4343
- Is the port open on the forwarding rule / is the forwarding rule configured
with
allPorts: true? - Got the VPC firewall rule created by the service operator in GKE updated to open all required ports?
- The Google Cloud Web Console is 100% broken and unable to deal with it. Don't touch it via the web.
- If you configure an
exceed_actionin agoogle_compute_security_policyterraform resource you must use a value with response code, e.g.exceed_action = "deny(429)". The response code will be ignored. In all other cases I know you must use adenywithout response code if you want to be able to assign the policy to a L3/L4 load balancer. - If you use config-connector (kcc) you can already use
exceedAction: denyalbeit it's not documented. Neither for config-connector itself nor for the API. - If you use the gcloud cli you can use
--exceed-action=denywhich is already documented if you callgcloud beta compute security-policies create --help, but it also works in the none beta mode. Also export / import via gcloud cli work with adenywithout defining a response code.
rule
description = "L3-L4 Rate Limit"
action = "rate_based_ban"
priority = "2342"
match
versioned_expr = "SRC_IPS_V1"
config
src_ip_ranges = ["*"]
rate_limit_options
enforce_on_key = "IP"
# exceed_action only supports deny() with a response code
exceed_action = "deny(429)"
rate_limit_threshold
count = 320
interval_sec = 60
ban_duration_sec = 240
ban_threshold
count = 320
interval_sec = 60
conform_action = "allow"
- action: rate_based_ban
description: L3-L4 Rate Limit
match:
config:
srcIpRanges:
- "*"
versionedExpr: SRC_IPS_V1
preview: false
priority: 2342
rateLimitOptions:
banDurationSec: 240
banThreshold:
count: 320
intervalSec: 60
conformAction: allow
enforceOnKey: IP
exceedAction: deny
rateLimitThreshold:
count: 320
intervalSec: 60
In the last past weeks, I have done something I had been meaning to do for a
while but always pushed back at the bottom of my TODO pile: prepare for my
death.
I am still quite young and perfectly healthy (mentally and physically) and I do
plan to live a long and full life, but death is something that comes from us all
and can strike anytime. Having witnessed friends and colleagues who lost loved
ones who did not prepare adequately for their passing, dealing with all this
legal stuff ahead of time seems like the best gift you can leave them.
Writing my will was the easiest part of this "preparation for death" process.
I have few material possessions and I'm leaving everything to my SO. As for the
copyright for my code, I have decided everything I wrote will be licensed under
Like each month, have a look at the work funded by 
My first impressions of the device are good - it is heavier than I anticipated,
but very vell made. It is very easy to assemble and disassemble and it feels
like it can take a hit.
When I started it the first time it took some minutes to boot because of the
new memory module, but then it told me right away that it could not detect an
operating system. As usual when I want to install a new system, I created
a 
I only learned later, that the SecureBoot setting is in a separate section
that is not part of the main BIOS configuration dialog. There is an Administer
Secure Boot icon which you can choose when starting the device, but
One small downside of the keyboard is that it does not have a backlight, which
was a surprise. But given that this is a frame.work laptop, there are chances
that a future generation of the keyboard will have backlight support.
The screen of the laptop can be turned all the way around to the back of the
laptops body, so it can be used as a tablet. In this mode the keyboard gets
disabled to prevent accidently pushing keys when using the device in tablet
mode.
For online meetings I still prefer using headphones with cables over bluetooth
once, so I m glad that the laptop has a headphone jack on the side.
Above the screen there are a camera and a microphone, which both have separate
physical switches to disable them.
I ordered a couple of expansion cards, in the current setup I use two USB-C,
one HDMI and one USB-A. I also ordered a 1TB expansion card and only used this
to transfer my