Node.js Buffer and TypedArray: Encodings and Binary Manipulation
A Buffer is a view over an ArrayBuffer, so byteLength is not string length and toString can silently corrupt split UTF-8. allocUnsafe and pooled buffers carry old data.

Quick Summary
A Buffer is a Uint8Array subclass that Node provides for binary data. It is a view over an ArrayBuffer, so buf.length is bytes, buf.byteOffset is where the view starts, and buf.buffer may be larger than the data — small buffers share an 8 KiB pool.
Encodings are the part that bites. A JS string is UTF-16, so str.length counts code units and Buffer.byteLength(str, 'utf8') counts bytes; those differ for anything outside ASCII. buf.toString() defaults to UTF-8 and silently replaces invalid byte sequences with U+FFFD, which is a data-loss bug, not an error.
Buffer.from(arrayBuffer) shares memory and Buffer.from(buffer) copies. When transferring to a worker thread, the shared pool means the underlying ArrayBuffer can contain other data, so an unsliced transfer leaks memory you did not intend to send.
Basic Syntax and Core Signatures
import { Buffer } from 'node:buffer'; // global, but explicit import is clearer
import { StringDecoder } from 'node:string_decoder';
// Create
const a = Buffer.alloc(16); // zero-filled
const b = Buffer.allocUnsafe(16); // NOT zeroed: old memory, security risk
const c = Buffer.from('héllo', 'utf8'); // 6 bytes: é is 2
const d = Buffer.from([0xde, 0xad, 0xbe, 0xef]);
const e = Buffer.from(new Uint8Array([1, 2, 3])); // copy
const f = Buffer.concat([Buffer.from('ab'), Buffer.from('cd')]);
const g = Buffer.from('48656c6c6f', 'hex');
// Convert
c.toString('utf8'); // 'héllo'
c.toString('hex'); // '68c3a96c6c6f'
d.toString('base64'); // '3q2+7w=='
d.toString('base64url'); // '3q2-7w' (URL-safe, no padding)
Buffer.byteLength('héllo', 'utf8'); // 6
Buffer.isBuffer(c); // true
// Bytes and views
const view = c.subarray(0, 2); // shares memory
c.length; c.byteOffset; c.buffer;
c[0]; // 0x68
const u8 = new Uint8Array(c.buffer, c.byteOffset, c.byteLength);
// Integers
const n = Buffer.alloc(8);
n.writeUInt32BE(0xdeadbeef, 0);
n.readUInt32BE(0); // 3735928559
n.readBigInt64LE(0);
n.writeDoubleBE(1.5, 0);
// Comparing
Buffer.compare(a, b); // -1 | 0 | 1
a.equals(b);
c.includes('llo');
// Streaming-decoding text
const dec = new StringDecoder('utf8');
dec.write(chunk1) + dec.write(chunk2) + dec.end();Key Comparison Table
| Expression | Copies | Shares memory | Use when |
|---|---|---|---|
Buffer.from(buf) | Yes | No | You need an independent copy |
Buffer.from(arrayBuffer) | No | Yes | Zero-copy over an existing buffer |
buf.subarray(a, b) | No | Yes | Slicing without allocation |
buf.slice(a, b) | No | Yes | Legacy alias of subarray |
new Uint8Array(buf) | Yes | No | Detaching from the 8 KiB pool |
Buffer.alloc(n) | — | Zeroed | Anything that will be sent or stored |
| Encoding | Bytes per unit | Round-trips arbitrary bytes | Notes |
|---|---|---|---|
utf8 | 1–4 | No, invalid sequences become U+FFFD | Default; text |
latin1 / binary | 1 | Lossy above 0xff | Byte-per-character mapping |
hex | 2 chars per byte | Yes | Hashes, debug dumps |
base64 | 4 chars per 3 bytes | Yes | Binary in JSON, with padding |
base64url | 4 chars per 3 bytes | Yes | URLs and JWT segments |
utf16le | 2 or 4 | No | Windows interop |
Practical Edge Cases and Deep Dives
1. byteLength is not length
const s = 'café';
s.length; // 4
Buffer.byteLength(s, 'utf8'); // 5
Buffer.from(s).length; // 5Any Content-Length, file offset, or chunk-splitting arithmetic on a string must use Buffer.byteLength, or a request body with an accented character is one byte short and the receiving parser hangs waiting for the rest. The same mistake in reverse — buf.length used as a character count — truncates a slice in the middle of a multi-byte sequence, producing U+FFFD at the boundary.
2. Splitting a UTF-8 stream corrupts text
// Wrong: a multi-byte character can straddle the chunk boundary
socket.on('data', (chunk) => process.stdout.write(chunk.toString()));
// Right
const dec = new StringDecoder('utf8');
socket.on('data', (chunk) => process.stdout.write(dec.write(chunk)));
socket.on('end', () => process.stdout.write(dec.end()));A 3-byte character split across two 64 KiB reads becomes two invalid sequences, and toString() on each half yields replacement characters. StringDecoder holds the incomplete tail until the next chunk. On the encoding side the same rule applies: never assume a chunk ends on a character boundary, and never split a buffer at an arbitrary index when the content is text.
3. allocUnsafe and the data that was already there
const buf = Buffer.allocUnsafe(1024); // may contain bytes from a previous allocationallocUnsafe is faster because it skips zeroing, and it is how pooled small buffers are carved out; the bytes it hands back can be remnants of something else the process handled — a decrypted payload, a token, part of another user's file. It is safe only when every byte is written before the buffer is read. For anything that is serialised, hashed, or sent over the wire, Buffer.alloc is the honest default, and the measurable cost is usually a rounding error next to the I/O that follows.
4. Transferring to a worker: the pool trap
import { Worker } from 'node:worker_threads';
const payload = Buffer.from('important data'); // may live in the 8 KiB pool
const copy = new Uint8Array(payload); // own ArrayBuffer
const worker = new Worker('./w.js', {
workerData: copy,
transferList: [copy.buffer],
});Transferring payload.buffer directly detaches the whole pool — including bytes of other buffers allocated nearby — and sends them to the worker. The bytes are not read by your code, but they were sent, which is a disclosure bug when one buffer holds a session token and another holds public data. new Uint8Array(payload) copies into a dedicated ArrayBuffer, and Buffer.from(payload) gives you a copy too. Also remember that after a transfer the source buffer is unusable (byteLength 0), and structuredClone(buf) returns a Uint8Array, not a Buffer.
5. Base64, padding, and URL safety
Buffer.from('user:pass').toString('base64'); // 'dXNlcjpwYXNz'
Buffer.from(token, 'base64url'); // decode, tolerant of missing padding
Buffer.from(emoji).toString('base64url'); // no '+', '/', or '=' to escapeStandard base64 uses + and /, which need URL encoding, and = padding, which some routers and signature schemes reject. base64url (Node 15.7+) implements RFC 4648 §5 and is what JWTs, signed URLs, and Authorization headers should use. Note that decoding a base64 string that was truncated is not an error — Node decodes what it can — so a token validation path must compare decoded bytes with timingSafeEqual, not strings.
6. Reading binary structure without a library
function parseHeader(buf) {
if (buf.length < 16) throw new Error('short header');
const magic = buf.toString('latin1', 0, 4);
const version = buf.readUInt16BE(4);
const flags = buf.readUInt8(6);
const size = buf.readBigUInt64BE(8);
return { magic, version, flags, size };
}read*/write* variants exist for UInt8, UInt16, UInt32, BigUInt64, Int*, Float, Double, in both LE and BE, with an offset that is bounds-checked (an out-of-range read throws ERR_OUT_OF_RANGE). Use latin1 for fixed-width magic strings so each byte maps to one character, and remember that readUInt32BE returns a Number that loses precision above 2^53 — use readBigUInt64BE for file sizes and identifiers from untrusted input.
Common Node.js Runtime Mistakes
| Symptom | Cause | Fix |
|---|---|---|
Content-Length off by a few bytes | str.length used instead of Buffer.byteLength | Count bytes |
| `` characters in streamed text | Chunk split inside a multi-byte character | StringDecoder |
Decoded token is null where a binary value was expected | Wrong encoding, e.g. hex on base64 data | Check the source encoding |
| Worker received unrelated data | Transferred a pooled ArrayBuffer | Copy with new Uint8Array(buf) |
TypeError: Buffer.from(number) | Deprecated numeric constructor removed | Buffer.alloc(n) |
ERR_OUT_OF_RANGE on readUInt32BE | Offset past the end of the buffer | Validate buf.length first |
ERR_INVALID_ARG_TYPE with a string | Passing text where a Buffer is required | Buffer.from(str) |
Base64 with + breaks a URL signature | Standard base64 used in a URL | base64url |
| Sensitive data in a heap dump | allocUnsafe used and not fully written | Buffer.alloc, buf.fill(0) after use |
slice mutated the original | subarray shares memory | Copy when the original must stay intact |
Try matching browser tools
All toolsSmall utilities that fit Node.js work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Node.js guides
All Node.js guidesNode.js EventEmitter: once, error Events, and Listener Memory Leaks
emit is synchronous, so a throwing listener aborts the rest. An error event with no listener crashes the process, and a listener per request is the classic leak.
Node.js fs/promises: Read, Write, Watch, and Atomic File Operations
writeFile truncates before writing, so a crash leaves half a file. The atomic pattern is temp file, fsync, rename; readFile loads everything, so large files need streams.
Node.js Streams: Readable, Writable, Transform, and pipeline
pipe() forwards data but not errors; pipeline() destroys the chain and rejects. Backpressure is the false return from write(), and _flush that never calls back hangs forever.
Node.js crypto: Hashing, HMAC, AES-GCM, and Random Bytes
Hashing, HMAC, AES-256-GCM and randomness solve different problems. Never reuse a GCM nonce, never hash a password with sha256, and compare secrets with timingSafeEqual.
More in Node.js
All Node.js guidesThe guides filed next to this one in Node.js, in library order.