Node.js Streams: Readable, Writable, Transform, and pipeline
pipe() forwards data but not errors; pipeline() destroys the chain and rejects. Backpressure is the false return from write(), and _flush that never calls back hangs forever.

Quick Summary
A stream is a sequence of chunks with flow control. Readable produces, Writable consumes, Duplex does both, and Transform rewrites as it passes through. Node's fs, net, http, and zlib APIs are all stream-based, so streaming is the default way to move data that does not fit in memory.
pipeline() from node:stream/promises is the correct way to connect them. It propagates errors, destroys every stream in the chain when one fails, and resolves only when the destination has finished. The older .pipe() does none of that: an error in the middle leaves the destination open and the source running.
Backpressure is what keeps memory flat. A Writable returns false from write() when its buffer is full, and a correct producer waits for 'drain' instead of writing until the heap grows. highWaterMark sets that buffer size — 64 KiB by default for byte streams in Node 22 and later, 16 KiB before it.
Basic Syntax and Core Signatures
import { Readable, Writable, Transform, PassThrough, pipeline, finished } from 'node:stream';
import { pipeline as pipelineP } from 'node:stream/promises'; // promises
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
// Composing
await pipelineP(
createReadStream('in.log'),
createGzip(),
createWriteStream('in.log.gz'),
);
// With cancellation
const ac = new AbortController();
await pipelineP(source, transform, dest, { signal: ac.signal });
// Readable from anything iterable
Readable.from(['a', 'b', 'c']);
Readable.from((async function* () { yield 1; yield 2; })());
Readable.from(Buffer.from('abc'));
// A custom Transform
class Upper extends Transform {
_transform(chunk, enc, cb) {
cb(null, chunk.toString().toUpperCase());
}
_flush(cb) {
cb(null, '\n');
}
}
// Backpressure by hand
async function writeAll(dest, chunks) {
for (const chunk of chunks) {
if (!dest.write(chunk)) await new Promise((r) => dest.once('drain', r));
}
await new Promise((r) => dest.end(r));
}// Web streams and Node streams convert both ways
import { Readable, Writable } from 'node:stream';
const web = Readable.toWeb(createReadStream('a.txt'));
const node = Readable.fromWeb(webBodyFromFetch);
await pipelineP(node, createWriteStream('b.txt'));Key Comparison Table
| API | Propagates errors | Destroys on failure | Waits for finish | Awaitable |
|---|---|---|---|---|
.pipe() | No | No | No | No |
pipeline() (callback) | Yes | Yes | Yes | No |
pipeline() (stream/promises) | Yes | Yes | Yes | Yes |
stream.compose() | Yes | Yes | Yes, for Duplex | No |
for await (chunk of rs) | Throws on error | Yes, if the loop exits | N/A | Yes |
| Stream | highWaterMark default (Node 22+) | Unit | Effect when exceeded |
|---|---|---|---|
Byte Readable/Writable | 64 KiB | Bytes | write() returns false, read pauses |
objectMode | 16 | Objects | Same, counted in objects |
fs.createReadStream | 64 KiB | Bytes | Limits chunk size |
http response | 16 KiB | Bytes | Socket flushes |
Practical Edge Cases and Deep Dives
1. Why .pipe() loses errors
const rs = createReadStream('missing.txt');
const ws = createWriteStream('out.txt');
rs.pipe(ws); // 'out.txt' is created and left open
rs.on('error', console.error); // without this: unhandled 'error' event, process exitspipe() connects data and end only. An error on the source is not forwarded, the destination is never ended, and the process throws Unhandled 'error' event if nothing listens. pipeline() forwards an AbortError or the original error to every participant, calls destroy() on each, and rejects its promise — which is why it is the only form worth writing in new code.
2. Backpressure that is silently ignored
for (const row of millionRows) {
if (!dest.write(row)) {
await new Promise((resolve) => dest.once('drain', resolve));
}
}Ignoring the false return value does not fail; it queues everything in memory and grows RSS until the process dies on a large export. The symptom is a memory curve that tracks input size, with no CPU spike. pipeline and for await (const chunk of readable) handle this for you — backpressure is automatic in a pull-based loop, because the loop body only runs when the previous chunk is consumed.
3. highWaterMark and chunk sizing
createReadStream('big.bin', { highWaterMark: 1024 * 1024 });
new Transform({ highWaterMark: 8, objectMode: true, transform(c, e, cb) { cb(null, c.x); } });The default is a latency/memory trade: 64 KiB means many small writes for a slow consumer, 1 MiB means fewer syscalls and more memory in flight. In objectMode the same number counts objects, so a highWaterMark of 16 is sixteen rows, not sixteen bytes. Setting it very low on a hot path costs throughput; setting it to Infinity disables backpressure entirely, which is only appropriate when the consumer is guaranteed faster than the producer.
4. A Transform that never ends
class Lines extends Transform {
_transform(chunk, enc, cb) {
this.buf = (this.buf ?? '') + chunk;
const parts = this.buf.split('\n');
this.buf = parts.pop();
for (const p of parts) this.push(p + '\n');
cb(); // no value: "more coming"
}
_flush(cb) {
if (this.buf) this.push(this.buf);
cb(); // must be called or 'finish' never fires
}
}Forgetting _flush leaves the last partial line in the buffer and, worse, if _flush is overridden but never calls cb, the writable side never finishes and pipeline hangs forever with no error. The same shape of bug appears when cb is called twice: Node throws ERR_MULTIPLE_CALLBACK, which is a lucky failure compared to the hang.
5. Streams, HTTP, and the client that disconnects
import { pipeline } from 'node:stream/promises';
import { createReadStream } from 'node:fs';
server.on('request', async (req, res) => {
try {
await pipeline(createReadStream(fileFor(req)), res);
} catch (err) {
if (err.code === 'ERR_STREAM_PREMATURE_CLOSE') return; // client went away
res.destroy(err);
}
});Piping to a response with pipeline closes the file descriptor when the client aborts mid-download — with .pipe(), the read stream keeps pulling the whole file into a socket nobody is reading. ERR_STREAM_PREMATURE_CLOSE is the expected signal for an aborted client and should not be logged as a server error. For uploads, req is the readable side; pipeline(req, createWriteStream(tmp)) gives you the same cleanup for free.
6. Object mode, PassThrough, and mixing units
const bus = new PassThrough({ objectMode: true });
bus.on('data', (obj) => console.log(obj.id));
bus.write({ id: 1 });A stream is either byte-oriented or object-oriented; writing a string to an objectMode stream sends the string itself as one object, and writing an object to a byte stream throws ERR_INVALID_ARG_TYPE. PassThrough is the standard way to expose an internal producer as a readable without buffering it yourself — but note that it buffers everything when nothing reads, so a PassThrough with no consumer is the same memory leak as an ignored write() return value.
For a stream you did not create — an HTTP response, a socket from a library — finished(stream) from node:stream/promises tells you when it is truly done, rejecting on error or premature close. Use it instead of listening for 'end' and 'finish' separately; 'end' only means the readable side saw EOF, while a writable side can still be flushing.
Common Node.js Runtime Mistakes
| Symptom | Cause | Fix |
|---|---|---|
Process exits on Unhandled 'error' event | .pipe() with no error listener | pipeline() |
| Destination file left at 0 bytes | Source errored, destination never ended | pipeline() |
| RSS grows with input size | write() result ignored | Await 'drain', or use pipeline |
pipeline hangs forever | _flush/cb never called, or called twice | Return cb() exactly once per call |
ERR_INVALID_ARG_TYPE writing objects | Byte stream got an object | objectMode: true |
ERR_STREAM_PREMATURE_CLOSE logged as an error | Client aborted the download | Treat as normal for HTTP |
| Chunk size smaller than expected | highWaterMark vs object count confusion | Set it in the right unit |
| Memory flat but throughput poor | highWaterMark too small, too many syscalls | Raise it, measure again |
write after end | Writing after end()/destroy() | Check writableEnded first |
Try matching browser tools
All toolsSmall utilities that fit Node.js work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Node.js guides
All Node.js guidesNode.js EventEmitter: once, error Events, and Listener Memory Leaks
emit is synchronous, so a throwing listener aborts the rest. An error event with no listener crashes the process, and a listener per request is the classic leak.
Node.js fs/promises: Read, Write, Watch, and Atomic File Operations
writeFile truncates before writing, so a crash leaves half a file. The atomic pattern is temp file, fsync, rename; readFile loads everything, so large files need streams.
Node.js crypto: Hashing, HMAC, AES-GCM, and Random Bytes
Hashing, HMAC, AES-256-GCM and randomness solve different problems. Never reuse a GCM nonce, never hash a password with sha256, and compare secrets with timingSafeEqual.
Node.js Buffer and TypedArray: Encodings and Binary Manipulation
A Buffer is a view over an ArrayBuffer, so byteLength is not string length and toString can silently corrupt split UTF-8. allocUnsafe and pooled buffers carry old data.
More in Node.js
All Node.js guidesThe guides filed next to this one in Node.js, in library order.