Garbage Collection Under Pressure: Memory Limits in Containers

Here’s a Go program dying in a 512 MiB container, as narrated by its own GC trace. The workload is deliberately boring — hold 260 MiB of live data, churn garbage 1 MiB at a time — and so is the setup: go1.26.4, a static linux/arm64 binary, alpine under podman with --memory=512m (cgroup v2, 6 Ps). Every number in this post comes from runs in that box.

gc 6 @0.027s 0%: ... 163->164->164 MB, 166 MB goal, 0 MB stacks, 0 MB globals, 6 P
gc 7 @0.142s 0%: ... 325->325->262 MB, 328 MB goal, 0 MB stacks, 0 MB globals, 6 P
[churn  250] heapAlloc=447MiB heapSys=451MiB sys=457MiB numGC=7
[churn  300] heapAlloc=497MiB heapSys=499MiB sys=505MiB numGC=7

Exit code 137 — and look at numGC, frozen at 7. Between that last collection and the kill, the heap climbed from 262 MiB to 497 MiB and the collector did nothing, because by its own arithmetic it didn’t have to: with 262 MB live after gc 7, the next collection wouldn’t trigger until the heap roughly doubled, to about 524 MiB (computed from the pacer formula — the trace never prints it, and this process never lived to see it). At its last stat line the runtime believed it had ~27 MiB of runway. The container’s ceiling was 512 MiB, and the kernel doesn’t negotiate.

Part 1 of this series covered how Go 1.25 finally made GOMAXPROCS read the cgroup CPU quota by default. Memory is the half of that story that hasn’t shipped — as of Go 1.26 the runtime still doesn’t look at memory.max, so two governors run the same process without ever speaking: a GC trigger that’s proportional to the live heap, and a kernel limit that’s absolute.

How Go decides when to collect

#

The GC guide gives the trigger formula exactly:

Target heap memory = Live heap + (Live heap + GC roots) * GOGC / 100

(GC roots — goroutine stacks and globals — are part of the formula “only as of Go 1.18”.) With the default GOGC=100 and a small root set, that reduces to: collect when the heap doubles past the last live set. Not a fixed budget, not a fraction of the machine — a ratio. You can watch it work locally with GODEBUG=gctrace=1, growing toward 200 MiB live:

gc 1 @0.001s 7%: ... 4->4->4 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 12 P
gc 4 @0.005s 4%: ... 36->36->36 MB, 36 MB goal, 0 MB stacks, 0 MB globals, 12 P
gc 7 @0.077s 0%: ... 277->277->202 MB, 280 MB goal, 0 MB stacks, 0 MB globals, 12 P

Goals walk 4 → 9 → 18 → 36 → 72 → 142 → 280 MB — textbook doubling while the program keeps everything alive. (The stacks and globals fields are those GC roots from the formula, printed since the 1.18 pacer rework; ~0 here.) The three numbers in 277->277->202 MB are heap size at trigger, at end of mark, and live afterward — and that last one seeds the next goal. On a laptop this schedule is a background detail; in a container it’s a countdown, computed by something that has no idea the walls exist.

How the kernel decides when to kill

#

Your Kubernetes resources.limits.memory becomes cgroup v2’s memory.max, and usage accrues in memory.current — which counts the whole cgroup: anonymous memory, page cache, kernel structures like dentries and inodes, TCP socket buffers, not just what your allocator asked for. The kernel’s cgroup-v2 documentation states the endgame plainly: “If a cgroup’s memory usage reaches this limit and can’t be reduced, the OOM killer is invoked in the cgroup.”

Note the “can’t be reduced” — the kernel reclaims what it can first (clean page cache gets squeezed before anyone dies), and only then kills. What it sends is SIGKILL, hence exit code 137: 128 + signal 9. No warning, and no handler… nothing your defer statements will ever see.

The collision, on tape

#

Back to the opening trace, now with the mechanism visible. In the container, goals walked 4 → 9 → 20 → 40 → 84 → 166 → 328 MB — same doubling shape, slightly different values because churn garbage rides along with the growing live set. Then gc 7, at 0.142 seconds into the run, finished with 262 MB live. Feed that into the formula and the next target lands at roughly 524 MiB — above the 512 MiB ceiling. From that moment the process was unkillable by its own GC and guaranteed to be killed by the kernel; it churned through another 235 MiB of garbage, sys climbed to 505 MiB, and the OOM killer closed the account.

Worth being precise about what this isn’t: not a leak, not a spike, not fragmentation. The runtime scheduled its next collection past the container limit because nothing ever told it there was a limit.

GOMEMLIMIT: the safety valve you must open yourself

#

Now the same binary, same container, one env var added — GOMEMLIMIT=410MiB, 80% of 512 MiB. (One honesty note: this run uses the harness’s churn mode with 2,000 iterations so it can terminate and show an exit code, where the OOMKill run used grow mode — same per-iteration allocation pattern, different run length.)

gc 7 @0.074s 0%: ... 293->293->262 MB, 296 MB goal, ... 6 P
gc 8 @0.270s 0%: ... 388->388->262 MB, 392 MB goal, ... 6 P
...
gc 22 @3.133s 0%: ... 388->388->262 MB, 392 MB goal, ... 6 P

This isn’t the folklore version (“the GC gets more aggressive as you approach the limit”). The mechanism is blunter: the limit lowers the goal. GOGC’s proportional math still wants ~524 MiB; the memory limit computes its own ceiling of ~392 MB (410 MiB minus the runtime’s non-heap memory), and the smaller number wins — every collection from gc 8 onward triggers at 388 MB, like a metronome. Twenty-two collections instead of seven, sys parked at 401 MiB, 2,000 MiB of garbage churned (roughly six times longer than the unprotected run survived), and then SURVIVED, exit 0.

GOMEMLIMIT shipped in Go 1.19, and it composes with GOGC rather than replacing it — the release notes point out it “will be respected even if GOGC=off”. The GC guide goes further: GOGC=off plus a memory limit “represents a maximization of resource economy” — which, in my book, retires the heap-ballast hack.

Here’s the part that stings after part 1. Go 1.25 made GOMAXPROCS cgroup-aware by default, but there’s no memory equivalent as of Go 1.26: the proposal to derive GOMEMLIMIT from the cgroup limit (golang/go#75164) is still open, and both the 1.25 and 1.26 release notes are silent on it. The community stopgap is automemlimit, which sets GOMEMLIMIT to 90% of the cgroup limit by default. Until the proposal lands, CPU configures itself and memory is your job.

Soft by design

#

The docs call GOMEMLIMIT a soft limit, and that word is doing two very different jobs — both demonstrable.

Face one: the limit yields rather than thrash. Retain 450 MiB live under the same 410 MiB limit — the live set alone exceeds the ceiling — and the runtime’s response is to overshoot (sys settles at 465 MiB, past the limit) while collecting almost continuously: 247 GCs in 0.36 seconds of trace, back-to-back, goals pinned right at the live set. What makes this survivable is a documented cap: GC CPU is limited to “roughly 50%, with a 2 * GOMAXPROCS CPU-second window”, so in the worst case “the program will slow down at most by 2x” instead of spiraling into pure collection. This run made progress the whole time and exited 0. Degraded, but alive.

Face two: it’s not a shield. Ask the same setup to retain 560 MiB — more than the container itself — and GOMEMLIMIT changes nothing that matters: 84 futile collections, killed at ~507 MB of heap, exit 137 anyway. A live set bigger than the box is a provisioning problem, and no env var fixes provisioning. A seatbelt, not a bigger car.

What GOMEMLIMIT counts — and what it can’t see

#

The SetMemoryLimit docs define the accounting: “all memory mapped, managed, and not released by the Go runtime” — effectively MemStats.Sys − HeapReleased. That includes goroutine stacks, which you can verify from runtime/metrics inside the container. With 2,000 goroutines parked on a channel:

/memory/classes/heap/objects:bytes                          65.3 MiB
/memory/classes/heap/stacks:bytes                           16.2 MiB
/memory/classes/total:bytes                                 92.9 MiB

That’s 2,000 default-sized stacks (~8.3 KiB apiece) sitting inside the total the limit governs — so no, you don’t need to budget stacks outside GOMEMLIMIT. What the docs exclude is what the runtime never touches: the binary’s own mappings, memory from C code via cgo, syscall.Mmap, and OS kernel memory held on behalf of the process.

The cgroup, meanwhile, counts things Go has no API for. Write a 300 MB file from inside the container:

before dd, memory.current: 946176
after 300MB file write, memory.current: 315764736
anon 53248
file 314572800

From 0.9 MB to 315.8 MB of charged memory — nearly all of it page cache — while the Go runtime’s view of the world didn’t move. The saving grace is that clean page cache is reclaimable: the kernel squeezes it before invoking the OOM killer (that “can’t be reduced” clause again), so cache alone rarely kills you. Dirty pages and tmpfs are the dangerous kind — charged the same way, and not droppable.

Budgeting: the 80% rule, honestly

#

How much headroom below the container limit does GOMEMLIMIT need? The wild offers three answers, and they’re not the same number:

• The GC guide says to “leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of” — and recommends the limit precisely for the container case.
• automemlimit ships 90% as its default.
• Folk practice says 80% — what the rescue run used (410 MiB on a 512 MiB box). No primary source exists for it; treat it as a conservative starting point, not a law.

The old version of this post claimed 5–10% of your memory goes to “allocator overhead”. Measured, that didn’t hold up: at a ~453 MiB heap, sys − heapAlloc was about 12 MiB — 2–3%, not 5–10 (workload-dependent; check /memory/classes/*). The real budget question isn’t allocator slack at all — it’s your non-runtime memory: page cache your I/O generates, cgo, mmap. That’s what the headroom is for, and it’s measurable, so measure it.

The two knobs stay complementary:

Set them as env vars, not in code, so you can tune without rebuilding:

env:
  - name: GOMEMLIMIT
    value: "410MiB"   # 80% of the 512MiB container limit — starting point
  - name: GOGC
    value: "100"

And the workflow, upgraded from vibes to measurement:

1. Baseline with GOGC=100, no limit, in a dev container.
2. Measure under real load: /memory/classes/* for the runtime’s side, memory.current for the cgroup’s. The gap is your invisible memory.
3. Budget: peak runtime memory + that gap + margin = container limit.
4. Protect: GOMEMLIMIT at 80–90% of the limit, adjusted by what step 2 told you.
5. Optimize: if GC CPU is too high, raise GOGC and let GOMEMLIMIT be the backstop.

Reproduce it yourself

#

The harness is under 150 lines — grow a package-level [][]byte to N MiB, churn 1 MiB chunks, print MemStats as you go:

GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o alloc-linux .
podman run --rm --memory=512m -v "$PWD":/app -e GODEBUG=gctrace=1 \
  docker.io/library/alpine:latest /app/alloc-linux -mode=grow -retain=260
# grow = run until killed; add -e GOMEMLIMIT=410MiB and -mode=churn
# for the rescue (exits 0 if it survives)

One gotcha that will silently falsify your results: keep the retained set in a package-level variable. In a local slice, liveness analysis lets the GC collect it after its last use — and your “rescue” run passes because the live set quietly vanished, not because the limit worked.

Opinionated Takeaway

#

Set GOMEMLIMIT on every containerized Go service you run, today, as an env var. Start at 80–90% of the container limit, then let measurements — not the ratio — pick the final number, because the right headroom is exactly your non-runtime memory and nothing else. Don’t wait for the runtime to do this for you: CPU got its cgroup-aware default in 1.25, memory’s proposal (#75164) is still open, and hope is not a pacer.

And keep the two failure modes straight, because they need different fixes. If numGC froze while the heap climbed to the kill line, that was a scheduling gap — GOMEMLIMIT solves it outright. If the GC was firing constantly and you still got 137, your live set doesn’t fit the box, and the fix is memory or architecture, not tuning… a seatbelt, not a bigger car.

There’s a bill for all this, though. The rescue traded 7 collections for 22, and the overshoot run collected 247 times in under half a second — GC frequency is latency you pay somewhere else, and Go 1.26’s new default collector (Green Tea) changes that arithmetic too. That’s next.

Next in the Go in Containers series: GC Pauses and Latency — what all that extra collecting costs you, and where the pauses actually hide.

Go in Containers series: Intro — A Silent Crisis · 1. GOMAXPROCS Demystified · 2. GC Under Pressure · 3. GC Pauses and Latency · 4. The Container Debugging Workflow