Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Go http.Client: unexpected EOF and Connection Reuse Leaks

A body closed without being read cannot be reused, and descriptors pile up. Idle timeouts shorter than the upstream's, MaxIdleConnsPerHost and safe retries.

~5 min readGo
#Go#HTTP#Performance#Debugging
3D illustration of a Go gopher-abstract cyan packet sliding along a connection rail that breaks into a severed end, a pile of socket rings below

Introduction & Quick Summary

Symptom. A Go service that calls an upstream API shows Get "https://api.example.com/v1/items": unexpected EOF or http: server closed idle connection a few times an hour, always on the first request after a quiet period. File descriptors climb towards the limit and dial tcp: lookup api.example.com: no such host appears once the process runs out. Under load, every request opens a new TCP connection even though keep-alive is on.

Cause. Two habits and one default. A resp.Body closed without being fully read leaves the transport unable to reuse its connection, because the unread remainder makes the stream state unknown, so the connection is dropped and, if the body is never closed at all, the file descriptor leaks. Separately, an upstream or load balancer closes idle connections after its own timeout, and the client's IdleConnTimeout (90 seconds by default) is longer, so the first reuse of a connection the server has already closed fails — the transport retries it only when the request is replayable. And MaxIdleConnsPerHost defaults to 2, so a client with high concurrency keeps two connections warm and dials the rest.

Fix. Always io.Copy(io.Discard, resp.Body) before Close() (or read to EOF), tune a cloned Transport with MaxIdleConnsPerHost, IdleConnTimeout, and ResponseHeaderTimeout set below the upstream's idle timeout, and treat one retry of an idempotent request as normal operation.

Core Mechanics, Signatures & Reproducible Setup

Connection reuse depends on the body being consumed to its end. Everything else in this article follows from that.

go
func get(ctx context.Context, client *http.Client, url string) ([]byte, error) { req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { return nil, err } resp, err := client.Do(req) if err != nil { return nil, err } defer resp.Body.Close() // drain first: without this the connection cannot be reused body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20)) if err != nil { return nil, err } return body, nil }

io.ReadAll reads to EOF, which is what the transport needs. The failure mode appears when the response is large and the caller stops early — a JSON decoder that stops after the first object, an if resp.StatusCode != 200 { return } that returns before touching the body.

go
// the leak: a non-200 response returned without reading anything if resp.StatusCode != http.StatusOK { return fmt.Errorf("upstream %d", resp.StatusCode) // body never drained } // the fix if resp.StatusCode != http.StatusOK { io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10)) return fmt.Errorf("upstream %d", resp.StatusCode) }

A client with sane limits, built once at startup:

go
func newClient() *http.Client { t := http.DefaultTransport.(*http.Transport).Clone() t.MaxIdleConns = 100 t.MaxIdleConnsPerHost = 20 // default is 2 t.IdleConnTimeout = 30 * time.Second // shorter than the upstream's 60s t.ResponseHeaderTimeout = 10 * time.Second t.TLSHandshakeTimeout = 5 * time.Second return &http.Client{Transport: t, Timeout: 30 * time.Second} }

Reproduce the idle-connection failure with a server that closes after two seconds and a client that waits three, then reuses. Go returns unexpected EOF on the first attempt and succeeds on the automatic retry for a GET; a POST with a body fails outright, because the transport will not replay a request whose body it has already consumed.

Key Comparison Table

SettingDefaultFailure when left alone
MaxIdleConnsPerHost2dial churn under concurrency
IdleConnTimeout90sreuse of connections the server already closed
MaxConnsPerHost0 (unlimited)no backpressure toward a struggling upstream
ResponseHeaderTimeout0 (none)goroutines stuck on a silent server
Client.Timeout0 (none)the whole request, body included, never expires
DisableKeepAlivesfalsesetting it true recreates every connection
Closing patternConnection reusedDescriptor freed
defer resp.Body.Close() onlyno, if body unreadyes
io.Copy(io.Discard, body) then Closeyesyes
Neithernono, leaks
ReadAll then Closeyesyes
Close on a 4xx without readingnoyes

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. unexpected EOF on a reused connection

The upstream closed the socket while the client still believed it was idle. Since Go 1.6 the transport retries when the request is idempotent and no response bytes arrived, so a GET usually succeeds transparently and the error surfaces only on POST, PATCH, or a request with a GetBody that is nil.

go
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload)) req.GetBody = func() (io.ReadCloser, error) { return io.NopCloser(bytes.NewReader(payload)), nil // enables safe retry }

With GetBody set, the transport may replay the request on a fresh connection. Without it, the error is final, and the correct client behaviour is to retry at the application layer with an idempotency key.

2. Leaked descriptors from early returns

Each unclosed body holds a connection and a file descriptor. A service doing 50 requests per second with a 5% early-return path leaks about 2.5 descriptors per second, which reaches ulimit -n in a couple of hours.

bash
# confirm the leak before changing code curl -s localhost:6060/debug/pprof/goroutine?debug=2 | grep -c "net/http.(*persistConn)" lsof -p $(pgrep -f myservice) | grep -c TCP

A rising count of persistConn goroutines that never falls is the signature. Goroutine profiles of a http.Client leak almost always point at a body that was never closed.

3. Client.Timeout also bounds the body read

Timeout covers dial, request write, response headers, and reading the body. A client with Timeout: 30 * time.Second reading a 2 GB download fails at thirty seconds even though the connection is healthy. For streaming, leave Timeout at zero and set ResponseHeaderTimeout on the transport plus a per-request context deadline.

go
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) defer cancel() req, _ := http.NewRequestWithContext(ctx, http.MethodGet, bigURL, nil)

4. A global client versus a client per request

http.Client is safe for concurrent use and holds the connection pool. Creating one per request discards the pool each time, so every call dials and handshakes.

go
// wrong func fetch(url string) (*http.Response, error) { return (&http.Client{}).Get(url) } // right var shared = newClient()

The same applies to http.DefaultClient, whose DefaultTransport has the two-idle-connection default and no timeouts at all — the reason http.Get is fine for a script and wrong for a service.

5. HTTP/2 and ForceAttemptHTTP2

A custom Transport built from scratch rather than cloned silently loses HTTP/2, because ForceAttemptHTTP2 is only true when the transport has the standard dialer and TLS config. The symptom is a client that negotiated h2 with the default transport and falls back to HTTP/1.1 after the custom transport is introduced.

go
t.ForceAttemptHTTP2 = true

6. Reading the connection state

httptrace answers the question the error message cannot: was this request reused or dialled?

go
trace := &httptrace.ClientTrace{ GotConn: func(info httptrace.GotConnInfo) { log.Printf("reused=%v idle=%v remote=%v", info.Reused, info.WasIdle, info.Conn.RemoteAddr()) }, } req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))

A reused=false on every request in steady state means the body is not being drained somewhere upstream of this call.

Common Compilation/Runtime Mistakes & Version Gotchas

  • defer resp.Body.Close() inside a loop. The defer runs at function exit, so N responses stay open until the loop ends. Wrap each iteration in a function.
  • Closing without draining. The connection is discarded rather than reused; it is a performance bug that looks like an upstream latency problem.
  • MaxIdleConnsPerHost left at 2. Under 100 concurrent requests, 98 connections are created and thrown away per burst.
  • IdleConnTimeout longer than the upstream's. Predictable unexpected EOF on the first request after idleness.
  • DisableKeepAlives: true "to fix" EOF errors. It multiplies handshakes and removes the retry that made GET reliable.
  • Custom Transport built literally. Loses HTTP/2, proxy from environment, and sensible dial timeouts.
  • Retrying a POST without an idempotency key. A replayed write creates duplicates; the safe retry is at the application layer with a key the server deduplicates on.

Related tools. Turn a curl -v transcript of the upstream into a Go client with cURL to code, inspect the JSON body of a failed response with the JSON formatter, and confirm two responses differ by more than formatting with the text diff.

Try matching browser tools

All tools

Small utilities that fit Go work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Go guides

More in Go

All Go guides

The guides filed next to this one in Go, in library order.