Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Node.js Streams: Readable, Writable, Transform, and pipeline

pipe() forwards data but not errors; pipeline() destroys the chain and rejects. Backpressure is the false return from write(), and _flush that never calls back hangs forever.

~4 min readNode.js
#Node.js#Streams#Backpressure#Performance
3D illustration of a pipeline with source tank, transform chamber and destination tank with pressure gauges

Quick Summary

A stream is a sequence of chunks with flow control. Readable produces, Writable consumes, Duplex does both, and Transform rewrites as it passes through. Node's fs, net, http, and zlib APIs are all stream-based, so streaming is the default way to move data that does not fit in memory.

pipeline() from node:stream/promises is the correct way to connect them. It propagates errors, destroys every stream in the chain when one fails, and resolves only when the destination has finished. The older .pipe() does none of that: an error in the middle leaves the destination open and the source running.

Backpressure is what keeps memory flat. A Writable returns false from write() when its buffer is full, and a correct producer waits for 'drain' instead of writing until the heap grows. highWaterMark sets that buffer size — 64 KiB by default for byte streams in Node 22 and later, 16 KiB before it.

Basic Syntax and Core Signatures

js
import { Readable, Writable, Transform, PassThrough, pipeline, finished } from 'node:stream'; import { pipeline as pipelineP } from 'node:stream/promises'; // promises import { createReadStream, createWriteStream } from 'node:fs'; import { createGzip } from 'node:zlib'; // Composing await pipelineP( createReadStream('in.log'), createGzip(), createWriteStream('in.log.gz'), ); // With cancellation const ac = new AbortController(); await pipelineP(source, transform, dest, { signal: ac.signal }); // Readable from anything iterable Readable.from(['a', 'b', 'c']); Readable.from((async function* () { yield 1; yield 2; })()); Readable.from(Buffer.from('abc')); // A custom Transform class Upper extends Transform { _transform(chunk, enc, cb) { cb(null, chunk.toString().toUpperCase()); } _flush(cb) { cb(null, '\n'); } } // Backpressure by hand async function writeAll(dest, chunks) { for (const chunk of chunks) { if (!dest.write(chunk)) await new Promise((r) => dest.once('drain', r)); } await new Promise((r) => dest.end(r)); }
js
// Web streams and Node streams convert both ways import { Readable, Writable } from 'node:stream'; const web = Readable.toWeb(createReadStream('a.txt')); const node = Readable.fromWeb(webBodyFromFetch); await pipelineP(node, createWriteStream('b.txt'));

Key Comparison Table

APIPropagates errorsDestroys on failureWaits for finishAwaitable
.pipe()NoNoNoNo
pipeline() (callback)YesYesYesNo
pipeline() (stream/promises)YesYesYesYes
stream.compose()YesYesYes, for DuplexNo
for await (chunk of rs)Throws on errorYes, if the loop exitsN/AYes
StreamhighWaterMark default (Node 22+)UnitEffect when exceeded
Byte Readable/Writable64 KiBByteswrite() returns false, read pauses
objectMode16ObjectsSame, counted in objects
fs.createReadStream64 KiBBytesLimits chunk size
http response16 KiBBytesSocket flushes

Practical Edge Cases and Deep Dives

1. Why .pipe() loses errors

js
const rs = createReadStream('missing.txt'); const ws = createWriteStream('out.txt'); rs.pipe(ws); // 'out.txt' is created and left open rs.on('error', console.error); // without this: unhandled 'error' event, process exits

pipe() connects data and end only. An error on the source is not forwarded, the destination is never ended, and the process throws Unhandled 'error' event if nothing listens. pipeline() forwards an AbortError or the original error to every participant, calls destroy() on each, and rejects its promise — which is why it is the only form worth writing in new code.

2. Backpressure that is silently ignored

js
for (const row of millionRows) { if (!dest.write(row)) { await new Promise((resolve) => dest.once('drain', resolve)); } }

Ignoring the false return value does not fail; it queues everything in memory and grows RSS until the process dies on a large export. The symptom is a memory curve that tracks input size, with no CPU spike. pipeline and for await (const chunk of readable) handle this for you — backpressure is automatic in a pull-based loop, because the loop body only runs when the previous chunk is consumed.

3. highWaterMark and chunk sizing

js
createReadStream('big.bin', { highWaterMark: 1024 * 1024 }); new Transform({ highWaterMark: 8, objectMode: true, transform(c, e, cb) { cb(null, c.x); } });

The default is a latency/memory trade: 64 KiB means many small writes for a slow consumer, 1 MiB means fewer syscalls and more memory in flight. In objectMode the same number counts objects, so a highWaterMark of 16 is sixteen rows, not sixteen bytes. Setting it very low on a hot path costs throughput; setting it to Infinity disables backpressure entirely, which is only appropriate when the consumer is guaranteed faster than the producer.

4. A Transform that never ends

js
class Lines extends Transform { _transform(chunk, enc, cb) { this.buf = (this.buf ?? '') + chunk; const parts = this.buf.split('\n'); this.buf = parts.pop(); for (const p of parts) this.push(p + '\n'); cb(); // no value: "more coming" } _flush(cb) { if (this.buf) this.push(this.buf); cb(); // must be called or 'finish' never fires } }

Forgetting _flush leaves the last partial line in the buffer and, worse, if _flush is overridden but never calls cb, the writable side never finishes and pipeline hangs forever with no error. The same shape of bug appears when cb is called twice: Node throws ERR_MULTIPLE_CALLBACK, which is a lucky failure compared to the hang.

5. Streams, HTTP, and the client that disconnects

js
import { pipeline } from 'node:stream/promises'; import { createReadStream } from 'node:fs'; server.on('request', async (req, res) => { try { await pipeline(createReadStream(fileFor(req)), res); } catch (err) { if (err.code === 'ERR_STREAM_PREMATURE_CLOSE') return; // client went away res.destroy(err); } });

Piping to a response with pipeline closes the file descriptor when the client aborts mid-download — with .pipe(), the read stream keeps pulling the whole file into a socket nobody is reading. ERR_STREAM_PREMATURE_CLOSE is the expected signal for an aborted client and should not be logged as a server error. For uploads, req is the readable side; pipeline(req, createWriteStream(tmp)) gives you the same cleanup for free.

6. Object mode, PassThrough, and mixing units

js
const bus = new PassThrough({ objectMode: true }); bus.on('data', (obj) => console.log(obj.id)); bus.write({ id: 1 });

A stream is either byte-oriented or object-oriented; writing a string to an objectMode stream sends the string itself as one object, and writing an object to a byte stream throws ERR_INVALID_ARG_TYPE. PassThrough is the standard way to expose an internal producer as a readable without buffering it yourself — but note that it buffers everything when nothing reads, so a PassThrough with no consumer is the same memory leak as an ignored write() return value.

For a stream you did not create — an HTTP response, a socket from a library — finished(stream) from node:stream/promises tells you when it is truly done, rejecting on error or premature close. Use it instead of listening for 'end' and 'finish' separately; 'end' only means the readable side saw EOF, while a writable side can still be flushing.

Common Node.js Runtime Mistakes

SymptomCauseFix
Process exits on Unhandled 'error' event.pipe() with no error listenerpipeline()
Destination file left at 0 bytesSource errored, destination never endedpipeline()
RSS grows with input sizewrite() result ignoredAwait 'drain', or use pipeline
pipeline hangs forever_flush/cb never called, or called twiceReturn cb() exactly once per call
ERR_INVALID_ARG_TYPE writing objectsByte stream got an objectobjectMode: true
ERR_STREAM_PREMATURE_CLOSE logged as an errorClient aborted the downloadTreat as normal for HTTP
Chunk size smaller than expectedhighWaterMark vs object count confusionSet it in the right unit
Memory flat but throughput poorhighWaterMark too small, too many syscallsRaise it, measure again
write after endWriting after end()/destroy()Check writableEnded first

Try matching browser tools

All tools

Small utilities that fit Node.js work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Node.js guides

More in Node.js

All Node.js guides

The guides filed next to this one in Node.js, in library order.