The goroutineleak Profile in Go 1.27: How to Catch Stuck Goroutines in ProductionArticle · digest for November 19
The goroutineleak Profile in Go 1.27: How to Catch Stuck Goroutines in Production
Article5 sections≈ 10 min

The goroutineleak Profile in Go 1.27: How to Catch Stuck Goroutines in Production

Your Go service runs smoothly for a week, all charts look green, and then memory quietly starts creeping up. The garbage collector keeps hitting GOMEMLIMIT, CPU usage climbs, and /debug/pprof/goroutine shows three thousand goroutines sitting there. Good luck guessing which ones are just waiting under load and which ones are stuck forever.

Go 1.27 solves this at the runtime level. The built-in goroutineleak profile, which was hidden behind GOEXPERIMENT=goroutineleakprofile in Go 1.26, has graduated from experimental status and is now on by default. Vlad Saioc broke it down on The Go Blog: the detector piggybacks on the garbage collector and points straight to the lines where goroutines are permanently stuck, with almost zero false positives.

Why Tests and Standard pprof Miss Quiet Leaks

A goroutine is a lightweight thread of execution managed by the Go runtime. Spawning them in batches is cheap, but if a goroutine sleeps on a channel or mutex and nothing will ever wake it up, you have a leak. A stuck goroutine never disappears. It holds onto its stack and every object its variables reference, forcing the garbage collector to scan that dead reference graph on every cycle and burn CPU for nothing.

Before Go 1.27, teams mostly caught these bugs in tests. uber-go/goleak checks after a unit test whether extra goroutines were left behind. The testing/synctest package from Go 1.25 lets you control concurrent ordering to reliably reproduce races. Both tools live in tests, though. In production, real traffic and timeouts create edge cases your test suite never saw.

Standard /debug/pprof/goroutine falls short in live services too. It lumps all blocked goroutines together without separating a permanent deadlock from normal waiting. A slow leak of one or two goroutines per hour can hide in that noise for years.

Catching Stuck Workers via /debug/pprof/goroutineleak

You don't need to change any application code to use the new profile. The goroutineleak type is built into runtime/pprof, and if your service already imports net/http/pprof, the /debug/pprof/goroutineleak endpoint appears automatically.

Let's test it on a classic bug from the Go blog (similar to issues found in Uber's production services). The processWorkItems function spawns workers and collects results from an unbuffered make(chan result) channel, where sending blocks until someone reads. On the first error, the function returns early:

package main

import (
    "errors"
    "log"
    "net/http"
    _ "net/http/pprof"
)

type workItem int
type workResult int
type result struct {
    res workResult
    err error
}

func processWorkItem(w workItem) (workResult, error) {
    if w <= 0 {
        return 0, errors.New("invalid work item")
    }
    return workResult(w * 2), nil
}

func processWorkItems(ws []workItem) ([]workResult, error) {
    // Unbuffered: on early return, workers block forever on send
    ch := make(chan result)
    for _, w := range ws {
        go func() {
            res, err := processWorkItem(w)
            ch <- result{res, err}
        }()
    }
    var out []workResult
    for range len(ws) {
        r := <-ch
        if r.err != nil {
            return nil, r.err
        }
        out = append(out, r.res)
    }
    return out, nil
}

func handleWork(w http.ResponseWriter, _ *http.Request) {
    items := []workItem{1, 2, 3, 4, 0, 6, 7, 8, 9, 10}
    if _, err := processWorkItems(items); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
    }
}

func main() {
    http.HandleFunc("/work", handleWork)
    log.Fatal(http.ListenAndServe("localhost:6060", nil))
}

When processWorkItems hits the zero item, it catches an error, returns early, and abandons the channel. The remaining workers reach ch <- result{res, err} and hang forever because nobody will ever read from ch again. Every request to /work leaves a fresh batch of "zombie" goroutines in memory.

Build the binary, run it, hit /work, and grab the profile with curl and go tool pprof:

go build -o leaky && ./leaky

# In another terminal, hit /work and capture the leak profile
curl http://localhost:6060/work
curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof
go tool pprof leak.prof

After a series of requests, running list processWorkItems inside the interactive pprof shell points straight to the offending line with zero clutter:

Type: goroutineleak
(pprof) list processWorkItems
Total: 116
ROUTINE ======================== main.processWorkItems.func1 in .../main.go
         0        116 (flat, cum)   100% of Total
         .          .     31:   go func() {
         .          .     32:       res, err := processWorkItem(w)
         .        116     33:       ch <- result{res, err}
         .          .     34:   }()

There are no background HTTP server goroutines or timers in the report—just the 116 stuck sends. The fix takes one line: give the channel a buffer matching the number of tasks.

// A buffer of len(ws) lets workers send their result and exit
ch := make(chan result, len(ws))

Now each worker drops its result into the buffer without blocking and exits cleanly, and the garbage collector later reclaims the channel.

Three Real Production Traps: range, Mutex, and Channel

In larger codebases, leaks also hide inside loops and sync primitives such as sync.Mutex and sync.WaitGroup. Here are three real-world patterns from the Go blog and open-source projects.

The first is an unclosed channel in for range. A range ch loop only exits after close(ch) is called and the buffer is drained. Forget to close the channel, and every worker waits forever:

func noCloseRange(list []any, workers int) {
    ch := make(chan any)
    for i := 0; i < workers; i++ {
        go func() {
            for item := range ch {
                _ = item
            }
        }()
    }
    for _, item := range list {
        ch <- item
    }
    // Without close(ch), workers hang forever on range ch
    close(ch)
}

There's a bonus trap here too: if someone passes workers == 0, the parent goroutine itself hangs on ch <- item. The goroutineleak profile catches that case as well.

The second case is a real bug from CockroachDB. A method acquires a sync.Mutex inside a loop, checks a flag, and exits with break without releasing the lock:

type Gossip struct {
    mu     sync.Mutex
    closed bool
}

func (g *Gossip) bootstrap() {
    for {
        g.mu.Lock()
        if g.closed {
            // Without Unlock(), the next caller hangs on Lock()
            g.mu.Unlock()
            break
        }
        g.mu.Unlock()
    }
}

The first call with closed: true exits while leaving the mutex locked, so every subsequent goroutine blocks on g.mu.Lock() forever. Nobody can unlock it anymore, and the profiler flags the exact Lock() line.

The third case is bug Moby28462 from the Docker engine. There, StateChanged held c.Lock() while sending on an unbuffered s.stop <- struct{}{}, while monitor() had already taken the default branch and was waiting on the same c.Lock(). They fixed the deadlock by closing the channel instead:

// close() never blocks, Lock() is released, and monitor()
// exits via the case <-stop branch
func (s *Health) CloseMonitorChannel() {
    if s.stop != nil {
        close(s.stop)
    }
}

Closing the channel completes immediately without waiting for a receiver, the mutex is released, and the background monitor() loop shuts down cleanly.

How the Runtime Separates Leaks from Normal Waiting

Instead of guessing based on timeouts, goroutineleak uses a strict reachability graph. The algorithm grew out of an ASPLOS 2025 paper by Aarhus University, Washington University in St. Louis, and Uber, and was refined for Go with Michael Knyszek, Michael Pratt, and PJ Malloy.

The core rule is inductive liveness: a goroutine is alive if it isn't blocked on a synchronization primitive, or if at least one primitive it's waiting on is reachable by pointers from another live goroutine. If no running goroutine can reach a channel or mutex, nobody left can wake the sleeping goroutine up.

The runtime uses its concurrent tri-color mark-and-sweep garbage collector (including Green Tea GC) and checks reachability in five steps:

  1. Initial mark roots include global variables and only unblocked goroutines, not all of them.
  2. The normal mark phase walks pointers from the roots and marks reachable memory.
  3. The runtime checks the remaining blocked goroutines: if a primitive a goroutine is waiting on was marked in step 2, that goroutine is declared alive, added to the roots, and marking repeats.
  4. Once no new live goroutines are found, every remaining blocked goroutine is marked leaked.
  5. A final mark pass runs with the leaked goroutines in the roots so the GC doesn't collect their memory. After the cycle, pprof captures stacks and keeps only the leaked ones.

Detector Limits and CPU Cost

Zero-false-positive accuracy comes with boundaries. First, the profile only tracks Go's built-in primitives: channel sends and receives (including nil channels), blocking select statements without default (including empty select {}), and sync.Mutex, sync.RWMutex, sync.WaitGroup, and sync.Cond. Hangs on network or file I/O, system calls, or custom spinlocks won't show up.

Second, memory overreach can hide a leak. If a reference to a forgotten channel stays in a global variable or a live goroutine, the GC treats it as reachable and stays quiet, even if nothing will ever write to it again. Keeping primitive scopes tight helps avoid this.

Finally, the detector only spots leaks that have already happened. Memory overhead is near zero, though in a worst-case daisy-chain scenario (G0 -> P1 -> G1 -> P2 -> G2 ...), marking serializes and checking blocked goroutines hits O(n²) per GC cycle. Because the collector runs concurrently with your service and leaked goroutines don't go away on their own, scraping /debug/pprof/goroutineleak once every four hours in production—alongside goleak and testing/synctest in tests—is plenty.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 4