GC Pauses and Latency: Where the Time Actually Hides

Here’s a Go program’s entire stop-the-world budget for churning through 2 GB of garbage on my laptop — 43 collection cycles, every pause accounted for, straight from runtime/metrics (/sched/pauses/total/gc:seconds):

gc cycles: 43
pauses p50=0.0205ms p90=0.0205ms p99=0.0492ms max=0.0492ms

Twenty microseconds median. Forty-nine microseconds worst case, across the whole run. That’s what Go’s two stop-the-world phases cost on a healthy box in go1.26.4, and it’s worth staring at, because the folklore about “millisecond GC pauses” is off by a factor of fifty — in Go’s favor.

Now the same collector, same machine, inside a container with --cpus=0.5:

GOMAXPROCS=2 NumCPU=6
gc cycles: 1088
pauses p50=0.0328ms p90=0.0492ms p99=83.9ms max=83.9ms

The median barely moved. The tail exploded by a factor of two and a half thousand. One number in this trace is doing all the screaming, and this post is about where it comes from, how to see it, and what actually fixes it — with every number from real runs, same harness style as part 2: go1.26.4, static linux/arm64 binaries, alpine under podman, cgroup v2.

The anatomy of a pause, with real names

#

Go’s GC stops the world exactly twice per cycle, and the runtime documentation names the phases precisely: “stop-the-world (STW) sweep termination, concurrent mark and scan, and STW mark termination.” (You’ll meet these names again in gctrace output, so learn them as the runtime spells them — the first pause flips the write barrier on, the second closes out the mark.)

Both phases are designed to be tiny, and the healthy-box histogram above shows the design working. But an STW pause has two components: the work, and the wait to be scheduled to do the work. The GC guide’s latency section lists five ways GC costs you latency — brief STW pauses are only the first, and on a healthy machine none of the five hurt.

In a container, the second component stops being negligible. The CFS quota is an average: --cpus=0.5 means 50ms of CPU time per 100ms period (kernel cgroup-v2 docs, cpu.max). Burn the budget early and the kernel freezes every thread in the cgroup until the next period. If that freeze lands while the runtime is inside an STW window — goroutines stopped, world halted, cleanup half-done — the pause is no longer “the work.” It’s the work plus the remainder of the CFS period.

The multiplication, on tape

#

The harness: busy-loop goroutines burning CPU (one per P) plus a 1 MiB/iteration allocation churn, 20 seconds, in a --cpus=0.5 container. Two runs, one variable.

Run A — Go 1.25+ defaults. The runtime reads the cgroup quota (part 1’s story) and picks its floor:

GOMAXPROCS=2 NumCPU=6
gc cycles: 1088
pauses p50=0.0328ms p90=0.0492ms p99=83.9ms max=83.9ms

Run B — GOMAXPROCS=4 forced, simulating the pre-1.25 world or any of the opt-outs part 1 catalogued:

GOMAXPROCS=4 NumCPU=6
gc cycles: 552
pauses p50=0.0983ms p90=0.197ms p99=101ms max=101ms

Look at Run B’s max: 101 milliseconds — one full CFS period plus the work it interrupted. That’s not a coincidence; that’s the mechanism photographed. The process was mid-STW when the quota ran dry, and the pause absorbed the entire kernel freeze. Every P you run above your quota makes the budget burn faster and the freeze more likely to land inside a pause window.

And note what Run A does not say: it does not say the 1.25 defaults save you. GOMAXPROCS=2 is the floor, a 0.5-CPU quota is a quarter of that floor’s appetite, and the tail still hit 84ms. Right-sized GOMAXPROCS narrows the window; only headroom (or tolerance) removes it. P50 tells you your GC is healthy. P99 tells you about your kernel. When p99/p50 is three orders of magnitude, you don’t have a GC problem — you have a scheduling problem wearing GC’s clothes.

Where the latency hides when it’s not hiding in pauses

#

The GC guide’s full list of latency sources, and this is the part most posts skip: STW pauses are item one of five. The others — the GC taking 25% of CPU during mark, user goroutines drafted into mark assists when allocation outpaces the collector, write-barrier overhead, and root-scanning delays for individual goroutines — cost you latency between the pauses, and under a CPU quota they cost you double: every CPU-second the GC spends is a CPU-second closer to the throttle.

That’s the invoice for part 2’s central trade, so let’s pay it explicitly. The GOMEMLIMIT rescue turned 7 collections into 22; the overshoot run collected 247 times in 0.36 seconds. Each of those extra cycles is ~25% CPU during its mark phase plus two STW windows plus assist pressure on your hottest allocators — all of it billed against the same quota your request handlers need. GC frequency is latency you pay somewhere else, and inside a container “somewhere else” is the CFS meter. When the meter runs out mid-pause, you get Run B.

Assists deserve one more sentence, because they’re the sneakiest of the five: when allocation outruns marking, the runtime makes the allocating goroutine do GC work at the allocation site. Your P99 spike may be a request goroutine paying a GC tax mid-request — invisible in pause histograms, visible only in an execution trace as MARK ASSIST slices.

Green Tea: the new arithmetic

#

Part 2 promised this. Go 1.26 made Green Tea the default garbage collector (release notes: “we expect somewhere between a 10–40% reduction in garbage collection overhead in real-world programs”), and you can still A/B it against the old collector with GOEXPERIMENT=nogreenteagc at build time — an escape hatch scheduled to disappear in 1.27.

The workload where it shines is exactly the one this series has been beating on: pointer-dense heaps. One million *int64 held live, ten forced cycles, both collectors, same box:

--- go1.26.4 default (Green Tea):
mode=ptrs n=1000000  wall/cycle=1.30ms  gc-cpu/cycle=4.57ms
  pauses: count=20 max<=0.0492ms
--- GOEXPERIMENT=nogreenteagc:
mode=ptrs n=1000000  wall/cycle=4.40ms  gc-cpu/cycle=5.38ms
  pauses: count=20 max<=0.0492ms

Read the two lines that matter. Wall time per cycle: 4.40ms → 1.30ms, a 3.4× faster mark, because Green Tea scans memory in contiguous blocks instead of chasing pointers object-by-object across the heap. STW pauses: unchanged — a few dozen microseconds under both collectors.

That’s the honest headline: Green Tea changes the throughput arithmetic, not the pause arithmetic. Your pauses were already microseconds; they still are. What shrinks is the concurrent mark phase — the 25%-CPU window, the assist pressure, the quota burn. In a container, that means fewer CPU-seconds on the CFS meter per cycle, which means fewer chances for the kernel to freeze you mid-anything. The multiplication effect gets less fuel, and part 2’s frequency tax gets 10–40% cheaper without touching a knob.

And pointer density is still your biggest lever

#

Same benchmark, Green Tea, one variable — a million int64 versus a million *int64:

mode=ints n=1000000  wall/cycle=0.10ms  gc-cpu/cycle=0.20ms
mode=ptrs n=1000000  wall/cycle=1.30ms  gc-cpu/cycle=4.57ms

Twenty-three times the GC CPU for the same million values, because a slice of a million integers is one object with no pointers — the GC guide is explicit that pointer-free memory is cheap to skip — while a million pointers is a million objects to chase. (Folklore says “1ms vs 100ms scan”; the measured gap on this box is 0.2ms vs 4.6ms per cycle. Directionally the same lesson, honestly smaller numbers.) If your heap is a pointer thicket — []*Thing, map[string]*Entry, linked structures — flattening it buys more than any GOGC value.

The rest of the allocation-rate toolbox is what it’s always been: sync.Pool for high-churn buffers, value slices over pointer slices, fewer tiny objects per request. Less garbage → fewer cycles → fewer STW windows for the kernel to land a freeze on → less assist pressure. Every arrow in that chain got measured above.

The tuning levers, ranked honestly

#

1. Give the GC less work (allocation rate, pointer density). The only lever that improves every row of the cost table at once. Measured above: 23× on GC CPU.

2. Give the runtime honest CPU accounting. Part 1’s whole story, now with a latency price tag: forced GOMAXPROCS=4 against a 0.5-CPU quota moved max pause from 84ms to 101ms and raised every percentile. Audit the opt-outs (runtime.GOMAXPROCS(NumCPU) in init code, base-image env vars).

3. Trade memory for frequency — with part 2’s caveat attached. Raising GOGC reduces cycle frequency, and fewer cycles means fewer throttle-exposed windows. But once GOMEMLIMIT’s ceiling is below GOGC’s proportional target, GOGC is dead weight — part 2’s metronome run (every cycle triggering at 388 MB, GOGC’s ~524 MiB wish ignored) is what that looks like. Near the limit, the limit is the pacer; raise GOGC there and nothing changes but your confidence.

4. The Kubernetes lever, split into its actual parts. (a) If throttling lands during GC windows, the quota is too tight for the runtime’s appetite — raise the limit or cut the appetite (levers 1–3). (b) request ≈ limit buys placement predictability — burst capacity above your request depends on node neighbors, so a big gap makes pause tails node-dependent and irreproducible. (c) The “remove CPU limits entirely, keep requests” school trades hard ceilings for scheduler fairness; it genuinely eliminates CFS-period freezes, at the cost of noisy-neighbor exposure. Whichever you choose, watch nr_throttled in the container’s cpu.stat (or container_cpu_cfs_throttled_seconds_total in cAdvisor/Prometheus) — if it’s climbing, your pauses are being amplified, whatever your dashboard’s average CPU says.

Measuring it yourself

#

• /sched/pauses/total/gc:seconds (runtime/metrics, Go 1.22+; the older /gc/pauses:seconds still exists but is deprecated — they’re the same histogram, I verified both report identical values). Track p50 and p99: p50 is your GC, p99 is your kernel.
• GODEBUG=gctrace=1 — the clock columns show wall time per phase; the phase names match the ones above.
• go tool trace — the only per-goroutine view: MARK ASSIST slices on request goroutines, and STW windows visibly stretching when the scheduler can’t run anything.
• cat /sys/fs/cgroup/cpu.stat inside the container — nr_throttled and throttled_usec are the kernel’s own confession, and correlating their growth with pause-histogram spikes is the diagnosis. (Part 4 turns this into a full workflow.)

The harness for every number in this post is ~150 lines total — a churn loop (with a 50 MB live ballast for realistic cycle spacing) plus busy-loop spinners for the container runs, and a two-mode slice benchmark for the pointer/Green Tea A/Bs — built with CGO_ENABLED=0 GOOS=linux, run under podman run --cpus=0.5 with plain -e GOMAXPROCS=... as the only variable. Rebuild it from the descriptions above in ten minutes; the shape matters, not the exact numbers — your box will print different ones, and even on mine the throttled tail bucket bounces between 42ms and 101ms run to run (83.9ms is the modal capture; the microsecond medians never move).

Opinionated Takeaway

#

• Stop repeating “millisecond pauses.” Healthy go1.26.4 pauses are tens of microseconds — 49µs max across 43 cycles on this box. If you’re seeing milliseconds, something is stretching them, and it’s almost never the collector.
• The tail is the kernel, not the GC. 101ms = one CFS period, photographed. Diagnose p99 pause spikes against cpu.stat’s nr_throttled before touching a single GC knob.
• Green Tea is a throughput gift, not a latency gift — 3.4× faster marks on pointer-heavy heaps, pauses unchanged. Take the 10–40% quota relief and say thank you, but don’t expect your histogram to move.
• Pointer density is the biggest lever you own — 23× GC CPU between []int64 and []*int64, measured. Flatten before you tune.
• GOGC near GOMEMLIMIT is a no-op — part 2’s metronome proved it; don’t budget latency wins from a knob the pacer is ignoring.
• GOMAXPROCS honesty is a latency feature. The 1.25 defaults narrowed my worst pause by 17ms without touching anything else — and an opt-out somewhere in your dependency tree silently un-narrows it.

The GC isn’t a language problem, and inside a container it isn’t even really a memory problem. It’s a resource-interaction problem: the collector spends CPU, the kernel meters CPU, and your P99 lives in the gap between their accounting. You now have the tools to read both ledgers. Next: the debugging workflow that puts them together when production is on fire.

Next in the Go in Containers series: The Container Debugging Workflow — pprof, ephemeral containers, and getting signals out of a scratch image at 3 a.m.

Go in Containers series: Intro — A Silent Crisis · 1. GOMAXPROCS Demystified · 2. GC Under Pressure · 3. GC Pauses and Latency · 4. The Container Debugging Workflow