A Go service on a Hetzner box was showing stop-the-world GC pauses of about 51 microseconds at the median. Then, a few times per hour, one pause hit 40 milliseconds. Same code, same heap, nothing in the application changed. The culprit was not the Go runtime at all. It was the Linux kernel deciding that the garbage collector’s metadata was the least valuable thing in memory and sending it to swap.
The full write-up is worth reading twice, because the failure it describes sits in a gap between two systems that are each behaving correctly.
Where the 40 milliseconds went
Go’s collector stops the world at two points, sweep termination and mark termination. During those pauses it reads runtime metadata that lives outside the managed heap, in a region the runtime never frees. That metadata is only touched during GC cycles, which makes it, from the kernel’s point of view, cold. Under memory pressure the kernel evicts pages by age, so GC bookkeeping pages go to swap first and get faulted back in exactly when the world is stopped.
The author wired up a bpf script to see inside one bad pause. Of the 40 milliseconds, 39 were spent in 228 major page faults along the do_swap_page path, inside GC bookkeeping functions like spanSet.reset, finishsweep_m and nextMarkBitArenaEpoch. A microsecond-scale pause had been multiplied by roughly 800 because the pages it needed were on an NVMe device instead of in RAM. Measurement artifacts and reproduction scripts are in the author’s go-gc-swap-cost repository.
Over 30 minutes the author counted 312 stop-the-world pauses, with the 40 millisecond spike appearing two or three times per memory spike. The Go 1.26 Green Tea collector did not change this materially; the metadata read pattern stayed the same.
There is a second, sneakier cost in the same write-up. Building a 511 KiB message normally takes 3 to 5 milliseconds per goroutine. With the relevant pages on NVMe it jumped to 105 milliseconds, and on a network volume to 903 milliseconds. That one hits the allocating goroutine only, so it hides from global pause metrics entirely.
Why this keeps happening to Go
This is at least the second round of Go versus the kernel’s memory management. Go 1.21.4 stopped marking heap memory with MADV_HUGEPAGE after issue 61718 showed that the default transparent huge page defrag setting could force unbounded direct reclaim and compaction stalls over 100 milliseconds. The runtime moved to MADV_COLLAPSE instead. Same shape of problem: the runtime assumes fast page operations, the kernel has other plans under pressure.
Swap adds a twist because it breaks an assumption on both sides. The GC assumes its metadata is resident and cheap to read. The kernel assumes least-recently-used pages are safe to evict. Both assumptions are locally reasonable. Composed, they produce the one thing a GC is never supposed to produce, a stop-the-world pause dominated by disk I/O.
Why do operators end up with swap on latency-sensitive boxes in the first place? Usually as OOM insurance. A service that spikes unpredictably gets a swap device so the kernel absorbs the spike instead of killing the process, and for batch or queue-heavy workloads that trade is fine. The write-up makes the counter-case for request-serving workloads: the spike gets absorbed as latency instead, at 800x magnification, in exactly the code path you cannot profile your way out of. Know which kind of workload you run before choosing.
One related runtime detail is worth knowing. Between Go 1.12 and 1.15 the runtime returned memory to the kernel with MADV_FREE, which made freed pages lazily reclaimable and left RSS looking inflated. Go 1.16 switched the default to MADV_DONTNEED so memory is released promptly and RSS means what it says, overridable with GODEBUG=madvdontneed. The switch matters here because prompt release keeps the process’s footprint honest, which is what lets a well-tuned GOMEMLIMIT keep the whole working set resident in the first place.
What operators can do
The blunt fix is to stop using swap to absorb Go memory spikes on latency-sensitive services. Either the working set fits in RAM, or the process should fail loudly. A swap-induced GC stall is far harder to diagnose than an OOM kill, and this write-up shows why: nothing in the application code path is wrong.
If you need a ceiling, GOMEMLIMIT is the better tool. Set it 5 to 10 percent below the cgroup limit so the runtime collects earlier and scavenges harder before the kernel starts reclaiming. A limit set too low causes GC thrashing instead of fast failure, so leave headroom. Pair it with GOMAXPROCS matched to the container, ideally via automemlimit and automaxprocs.
THP policy matters too. The GC guide recommends transparent_hugepage/defrag set to defer or defer+madvise, and khugepaged/max_ptes_none at 0, for production Linux in constrained environments.
Finally, instrument for the failure mode before it happens. GODEBUG=gctrace=1 shows pause distribution, and page-fault accounting during STW windows via bpf is the only way to see this class of stall, because heap metrics will look perfectly healthy while the pause counter lies.
The general lesson crosses abstraction boundaries: a runtime’s tail latency is a property of the host kernel configuration as much as of the runtime itself. Treating swappiness, THP and swap devices as part of the Go deployment surface is the difference between a service that glitches twice an hour and one that does not.