go-retryablehttp: why resending a body doesn't guarantee a safe operationNote · digest for November 3
Object still life with a pair of analog counters and a transaction key illustrating idempotence of network requests in Go.
Note2 sections≈ 5 min

go-retryablehttp: why resending a body doesn't guarantee a safe operation

Every Go backend developer eventually runs into the whims of distributed networks: a brief TCP connection drop, an edge proxy timeout, or a Kubernetes pod restart. In the standard net/http client, retrying a failed request isn't that simple: the req.Body stream is read to EOF and closed on the very first attempt, and a second call fails with http: Request.Body already consumed.

To avoid reinventing byte-buffering wheels, the community usually grabs HashiCorp's battle-tested go-retryablehttp library. It carefully wraps the standard transport, knows how to rewind an io.ReadSeeker, and automatically retries on 500, 502, 503, and 504 status codes.

Yet a dangerous illusion lurks here: transport retryability has nothing to do with application idempotence.

The RetryMax trap and dropped responses

The library's first non-obvious rake lies in attempt arithmetic. When you set retryClient.RetryMax = 2 in configuration, it's easy to assume the service will make two requests total. In reality, the parameter means the number of retries after the initial failed attempt. That means the client will make up to three requests: one initial plus two retries. If an external API bills per request or has strict rate limits, that extra attempt can be an unpleasant surprise.

The second problem is far more serious and concerns mutating methods (POST, PATCH). Picture a payment charge or order creation scenario:

  1. The client sends a POST request with the order body to a payment gateway server.
  2. The gateway validates the request, deducts the balance, and commits the transaction to the database.
  3. Just as the 200 OK response is being sent, the link drops or the load balancer resets the connection on timeout.
  4. go-retryablehttp sees a connection reset by peer network error, rewinds the request body buffer, and dutifully sends it a second time.

The outcome is predictable: if the API on the other side isn't built to track duplicates, the user gets charged twice. Retrying blindly at the transport layer is safe only for HTTP-idempotent methods (GET, HEAD, PUT, DELETE). For mutating operations, an application-level Idempotency-Key header with a unique operation UUID is mandatory:

// Generate the idempotency header before sending
req, err := retryablehttp.NewRequestWithContext(ctx, "POST", apiURL, bytes.NewReader(payload))
if err != nil {
    return err
}

// A unique identifier ensures the gateway returns a cached response instead of duplicating
req.Header.Set("Idempotency-Key", orderUUID.String())
req.Header.Set("Content-Type", "application/json")

Custom CheckRetry policy and random jitter

Another nuance people often forget is backoff behavior. The standard DefaultBackoff delay function in go-retryablehttp faithfully grows the pause exponentially, but adds no random jitter. If an external microservice blinks briefly, hundreds of parallel instances of your app will retry in lockstep, creating the classic thundering herd problem and finishing off the struggling backend.

Proper client initialization requires overriding the backoff function with a random offset and watching for context cancellation:

package main

import (
    "context"
    "math/rand"
    "net/http"
    "time"

    "github.com/hashicorp/go-retryablehttp"
)

func createSafeClient() *http.Client {
    retryClient := retryablehttp.NewClient()
    retryClient.RetryMax = 2 // 1 initial + 2 retries
    retryClient.RetryWaitMin = 100 * time.Millisecond
    retryClient.RetryWaitMax = 2 * time.Second

    // Add random jitter up to 25% of the interval
    retryClient.Backoff = func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration {
        mult := time.Duration(1 << uint(attemptNum))
        sleep := min * mult
        if sleep > max {
            sleep = max
        }
        jitter := time.Duration(rand.Int63n(int64(sleep / 4)))
        return sleep + jitter
    }

    // Context cancellation immediately stops the retry loop
    retryClient.CheckRetry = func(ctx context.Context, resp *http.Response, err error) (bool, error) {
        if ctx.Err() != nil {
            return false, ctx.Err()
        }
        return retryablehttp.DefaultRetryPolicy(ctx, resp, err)
    }

    return retryClient.StandardClient()
}
RAM consumption

Keep in mind that for non-rewindable io.Reader interfaces, the library reads the entire request body into a byte slice in RAM. If files hundreds of megabytes in size are sent through go-retryablehttp, this will cause a sharp spike in RAM usage and GC pressure.

The bottom line: go-retryablehttp brilliantly takes care of transport boilerplate, but architectural responsibility for data idempotence always stays with the developer.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 3