The Container Debugging Workflow: A Practical Guide

You’ve read the theory. You know the GOMAXPROCS rounding logic, the GOMEMLIMIT safety valve, and where the pauses actually hide. But now you’re staring at a production alert: Pod X is restarting every 4 hours with Exit Code 137.

The evidence is split across three layers — the Go runtime, the container engine, and the Linux kernel — and debugging is the art of correlating them. This post is the workflow, and in keeping with the rest of the series, every command in it was actually run: go1.26.4, a static binary in a FROM scratch image, podman, cgroup v2. Where Kubernetes differs from my podman box, I say so.

Step 0: Get a signal out of a shell-less container

#

Here’s the wall you hit first, and most guides pretend it isn’t there. Production Go images are scratch or distroless — no shell, no coreutils. The classic first move fails before you start:

$ podman exec dbgdemo cat /sys/fs/cgroup/cpu.max
Error: crun: executable file `cat` not found in $PATH: No such file
or directory: OCI runtime attempted to invoke a command that was not found
exit: 127

No cat, no sh, no anything. You have three ways out, and you should set up the first two before the incident:

1. Make the binary confess at startup. One log line, and every restart documents the runtime’s world-view:

log.Printf("GOMAXPROCS=%d NumCPU=%d", runtime.GOMAXPROCS(0), runtime.NumCPU())

From my box, started with --cpus=1:

2026/07/05 21:49:35 GOMAXPROCS=2 NumCPU=6

That single line already confirms part 1’s floor rule (quota 1 → max(⌈1⌉, 2) = 2) with zero exec access. If this line says GOMAXPROCS=32 under a 1-CPU limit, you’re not looking at a cgroup bug — you’re looking at a pre-1.25 toolchain or one of part 1’s silent opt-outs (runtime.GOMAXPROCS(runtime.NumCPU()) in someone’s init code, a base-image env var). Grep your dependency tree, not the kernel.

2. SIGQUIT — the debugger that’s always installed. The Go runtime itself responds to SIGQUIT by dumping every goroutine’s stack and exiting (runtime docs, GOTRACEBACK). No shell needed — send the signal from outside:

$ podman kill --signal=QUIT dbgdemo
$ podman logs dbgdemo
SIGQUIT: quit
PC=0x9e144 m=0 sigcode=0

goroutine 9 gp=0x6b89eb964000 m=0 mp=0x9b7f60 [running]:
runtime.memclrNoHeapPointers()
	/opt/homebrew/Cellar/go/1.26.4/libexec/src/runtime/memclr_arm64.s:180 ...
runtime.mallocgc(0x100000, 0x46d860, 0x1)
runtime.makeslice(0x46d860?, 0x100000?, 0x100000?)
main.churn(...)

The full dump — every goroutine, every stack, delivered to the container log (note the podman logs; the kill itself prints nothing) — from a scratch image containing exactly one file. And look what it caught: the churn goroutine [running], mid-makeslice, zeroing a fresh 0x100000-byte buffer. One signal, and the allocation hot loop is on tape with a line number. It kills the process (exit 2), so it’s a last resort on a healthy pod and a free autopsy on a wedged one. On Kubernetes: kubectl exec won’t work without a shell, and there’s no kubectl-native way to send SIGQUIT — pod deletion only ever sends SIGTERM, then SIGKILL. What does work —

3. Ephemeral debug containers. kubectl debug -it <pod> --image=busybox:1.36 --target=<container> attaches a toolbox container to the running pod’s PID namespace (Kubernetes docs — GA since 1.25; --target shares the process namespace so you can see the Go process). Now you have a shell next to the shell-less container, and cat /sys/fs/cgroup/cpu.max works from inside it. I don’t have a cluster on this box, so unlike everything else here that command is quoted from the docs, not from a run — the podman equivalents below are the parts I can vouch for personally.

Step 1: The quick glance — and what 137 actually proves

#

Exit code 137 is 128 + 9: SIGKILL. The OOM killer sends SIGKILL — but so does the kubelet on eviction, a failed liveness probe’s restart escalation, and docker stop after its grace timeout. 137 is a hint. The confirmation is the container state, which the engine records:

$ podman inspect dbgdemo --format 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'
exited exit=137 oom=true

That oom=true (on Kubernetes: kubectl describe pod → Last State: Terminated, Reason: OOMKilled) is the actual verdict — and yes, that’s a real OOMKill: my demo app churning 1 MiB allocations under --memory=512m with no GOMEMLIMIT, dead in under a second, exactly the part-2 failure mode.

Restart rhythm is the other free signal: every few hours at steady traffic smells like a leak or slow growth toward the limit; restarts that track traffic spikes smell like capacity. Both get diagnosed the same way — keep reading.

Step 2: Validate the runtime’s world-view

#

Two numbers, two checks, both doable without exec:

• GOMAXPROCS — the startup log line above, checked against the limit. Mismatch = opt-out hunt (part 1 has the full list).
• GOMEMLIMIT — and here’s a dashboard trick that costs nothing. The default Prometheus Go collector (client_golang) already exports it:

go_gc_gomemlimit_bytes 9.223372036854776e+18

That’s a real scrape from my demo. 9.22e18 is math.MaxInt64 — the “unset” sentinel. If your dashboard shows that number, GOMEMLIMIT isn’t configured, and part 2 told you what happens next. A configured pod shows the actual byte value; alert on the sentinel, it’s the cheapest misconfiguration detector you’ll ever deploy.

Step 3: pprof — with the two mistakes everyone makes

#

Expose pprof deliberately: import _ "net/http/pprof" and serve it on a separate, localhost-bound port — never on your service listener; these endpoints leak heap contents and can be DoS’d. Reach it via kubectl port-forward (or from an ephemeral debug container).

import _ "net/http/pprof"
go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }()

Mistake 1: reading the heap profile as an allocation profile. Here’s my churn demo — a goroutine allocating 1 MiB buffers in a hot loop — through the default heap view:

$ go tool pprof -top http://localhost:6060/debug/pprof/heap
Type: inuse_space
      flat  flat%   sum%        cum   cum%
  224.36MB   100%   100%   224.36MB   100%  main.churn

Two things this output settles. Heap profiles attribute memory to your call sites — main.churn, named and shamed; runtime.mallocgc doesn’t even make the table — so if a guide tells you to look for “growth in mallocgc” in a heap profile, it’s confusing it with a CPU profile. And inuse_space shows what’s retained right now — the 218 MB my demo deliberately holds, plus a few MiB of fresh churn the sweeper hasn’t returned yet. The terabytes of short-lived garbage it churned are invisible here. They live in the allocation view:

$ go tool pprof -sample_index=alloc_space -top http://localhost:6060/debug/pprof/allocs
Type: alloc_space
   18.63TB   100%   100%    18.63TB   100%  main.churn

18.63 terabytes allocated lifetime — this snapshot came after about two minutes of churn — versus 224 MB in use. That ratio is what “you’re churning” looks like. Leak hunts use inuse_space and compare two snapshots over time; GC-pressure hunts use alloc_space. Same endpoint family, different questions.

Goroutine leaks are a third question — stacks live outside the heap profile entirely. If go_goroutines climbs without bound, capture /debug/pprof/goroutine?debug=1 and look for thousands of goroutines parked on the same source line.

And the RSS-vs-heap gap: if pprof shows a small heap but the cgroup shows huge usage, don’t jump to “cgo leak.” Part 2 measured a 300 MB file write moving memory.current by 315 MB while the Go runtime saw nothing — page cache, tmpfs, syscall.Mmap, and goroutine stacks are all suspects the heap profile can’t see. Cgo is on the list; it isn’t the list.

Mistake 2: hunting throttling in a CPU profile. A CPU profile from my churn demo under Green Tea (go1.26.4):

$ go tool pprof -top -seconds 5 http://localhost:6060/debug/pprof/profile
Duration: 5s, Total samples = 4.99s (99.80%)
      flat  flat%   sum%        cum   cum%
     3.64s 72.95% 72.95%      3.64s 72.95%  runtime.memclrNoHeapPointers
     0.04s   0.8% 88.58%      0.13s  2.61%  runtime.scanObject
     0.02s   0.4% 90.38%      4.23s 84.77%  main.churn
     0.01s   0.2% 92.99%      4.21s 84.37%  runtime.mallocgc
         0     0% 93.39%      0.55s 11.02%  runtime.gcBgMarkWorker

(Rows excerpted.) Read the runtime’s share: the mark workers (gcBgMarkWorker → gcDrain — running as gcDrainMarkWorkerFractional, because 25% of GOMAXPROCS=2 doesn’t earn a dedicated worker) take ~11%, Green Tea’s scanObject 2.6% — cheap here because 1 MiB byte slices are pointer-free, exactly part 3’s point — and the real bill is mallocgc at 84% cum, almost all of it memclrNoHeapPointers zeroing fresh buffers. That’s the collector and allocator spending your quota — allocation pressure, part 2’s territory. What this profile cannot show you is throttling: pprof samples on-CPU time, and time the kernel freezes you is off-CPU and invisible. The throttling verdict comes from the kernel’s own ledger, which brings us to —

Step 4: The kernel’s confession

#

From inside the container’s cgroup (via ephemeral container on k8s; directly on podman):

$ cat /sys/fs/cgroup/cpu.stat        # during a --cpus=0.5 run of the churn demo
nr_periods 123
nr_throttled 122
throttled_usec 2430155

Throttled in 122 of 123 periods — 2.4 seconds confiscated in a 12-second window. That’s the same starvation mechanism that produced part 3’s 84ms pause tails, seen from the kernel’s side of the desk. In Kubernetes you get the same ledger continuously as cAdvisor’s container_cpu_cfs_throttled_seconds_total — if its rate is nonzero while your pause histogram spikes, the correlation is the diagnosis.

Step 5: The dashboard that catches all of this while you sleep

#

All from the default client_golang collector (verified scrape) plus cAdvisor — no custom instrumentation:

Two corrections to the folk version of this table. Compare GOMEMLIMIT against go_memstats_sys_bytes, not heap_alloc — the limit governs total runtime memory (part 2 measured the ~12 MiB gap between them, which is exactly the margin that folk alert erases). And a P99 GC “duration” of 50ms isn’t a GC problem — healthy pauses are microseconds (part 3, measured); double-digit milliseconds mean the kernel stretched a pause, so check the throttling counter first.

The on-call checklist

#

1. Confirm the kill: engine state (oom=true / Reason: OOMKilled), not just exit 137.
2. Read the startup line: GOMAXPROCS vs limit. Mismatch → opt-out hunt, not cgroup archaeology.
3. Check the sentinel: go_gc_gomemlimit_bytes = 9.22e18 means unset → part 2, set it (80–90% of the limit as the starting point, then measure the non-runtime gap).
4. Throttling counter climbing → part 3; fix quota/GOMAXPROCS/allocation rate before touching GC knobs.
5. Leak triage: sys_bytes trend for the runtime, two inuse_space snapshots for the culprit, alloc_space for churn, goroutine profile for parked armies, and remember the runtime can’t see page cache.
6. Wedged, shell-less, out of options: SIGQUIT. The autopsy is free.

Opinionated Takeaway

#

• Instrument before the incident. The startup log line, the localhost pprof port, and the default Prometheus collector cost ~five lines total and turn every future 3 a.m. into a lookup instead of an expedition.
• 137 is a hint; OOMKilled is a verdict. Always read the state the engine recorded.
• Heap ≠ allocations. inuse_space for leaks, alloc_space for churn — and your code is in the top row, not mallocgc.
• pprof can’t see throttling. On-CPU samples only. The kernel’s cpu.stat is the ledger that can’t lie — 122/123 periods, on tape above.
• Alert on the 9.22e18 sentinel. The cheapest GOMEMLIMIT audit in existence.
• SIGQUIT is the debugger that ships in every image, including FROM scratch. Nothing to install; everything to read.

Debugging Go in containers is correlation: pause spikes × throttle counter = quota problem; OOMKill × unset sentinel = budgeting problem; big cgroup usage × small heap = look outside the runtime. The runtime, the engine, and the kernel each keep honest books — the workflow is just reading all three.

That closes the series. A Go binary in a container is three systems pretending to be one, and all four posts were about the seams: CPU accounting (part 1), memory accounting (part 2), the latency bill for both (part 3), and now the toolbox for reading the seams in production. The vault-level lesson is the same every time — the defaults got smarter, but they can’t see everything, and the gap between what the runtime assumes and what the kernel enforces is where your pager lives.

Go in Containers series: Intro — A Silent Crisis · 1. GOMAXPROCS Demystified · 2. GC Under Pressure · 3. GC Pauses and Latency · 4. The Container Debugging Workflow