Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Node.js Buffer and TypedArray: Encodings and Binary Manipulation

A Buffer is a view over an ArrayBuffer, so byteLength is not string length and toString can silently corrupt split UTF-8. allocUnsafe and pooled buffers carry old data.

~4 min readNode.js
#Node.js#Buffer#Binary#Encoding
3D illustration of a byte grid slab with encoding tiles flipping between hex and base64

Quick Summary

A Buffer is a Uint8Array subclass that Node provides for binary data. It is a view over an ArrayBuffer, so buf.length is bytes, buf.byteOffset is where the view starts, and buf.buffer may be larger than the data — small buffers share an 8 KiB pool.

Encodings are the part that bites. A JS string is UTF-16, so str.length counts code units and Buffer.byteLength(str, 'utf8') counts bytes; those differ for anything outside ASCII. buf.toString() defaults to UTF-8 and silently replaces invalid byte sequences with U+FFFD, which is a data-loss bug, not an error.

Buffer.from(arrayBuffer) shares memory and Buffer.from(buffer) copies. When transferring to a worker thread, the shared pool means the underlying ArrayBuffer can contain other data, so an unsliced transfer leaks memory you did not intend to send.

Basic Syntax and Core Signatures

js
import { Buffer } from 'node:buffer'; // global, but explicit import is clearer import { StringDecoder } from 'node:string_decoder'; // Create const a = Buffer.alloc(16); // zero-filled const b = Buffer.allocUnsafe(16); // NOT zeroed: old memory, security risk const c = Buffer.from('héllo', 'utf8'); // 6 bytes: é is 2 const d = Buffer.from([0xde, 0xad, 0xbe, 0xef]); const e = Buffer.from(new Uint8Array([1, 2, 3])); // copy const f = Buffer.concat([Buffer.from('ab'), Buffer.from('cd')]); const g = Buffer.from('48656c6c6f', 'hex'); // Convert c.toString('utf8'); // 'héllo' c.toString('hex'); // '68c3a96c6c6f' d.toString('base64'); // '3q2+7w==' d.toString('base64url'); // '3q2-7w' (URL-safe, no padding) Buffer.byteLength('héllo', 'utf8'); // 6 Buffer.isBuffer(c); // true // Bytes and views const view = c.subarray(0, 2); // shares memory c.length; c.byteOffset; c.buffer; c[0]; // 0x68 const u8 = new Uint8Array(c.buffer, c.byteOffset, c.byteLength); // Integers const n = Buffer.alloc(8); n.writeUInt32BE(0xdeadbeef, 0); n.readUInt32BE(0); // 3735928559 n.readBigInt64LE(0); n.writeDoubleBE(1.5, 0); // Comparing Buffer.compare(a, b); // -1 | 0 | 1 a.equals(b); c.includes('llo'); // Streaming-decoding text const dec = new StringDecoder('utf8'); dec.write(chunk1) + dec.write(chunk2) + dec.end();

Key Comparison Table

ExpressionCopiesShares memoryUse when
Buffer.from(buf)YesNoYou need an independent copy
Buffer.from(arrayBuffer)NoYesZero-copy over an existing buffer
buf.subarray(a, b)NoYesSlicing without allocation
buf.slice(a, b)NoYesLegacy alias of subarray
new Uint8Array(buf)YesNoDetaching from the 8 KiB pool
Buffer.alloc(n)—ZeroedAnything that will be sent or stored
EncodingBytes per unitRound-trips arbitrary bytesNotes
utf81–4No, invalid sequences become U+FFFDDefault; text
latin1 / binary1Lossy above 0xffByte-per-character mapping
hex2 chars per byteYesHashes, debug dumps
base644 chars per 3 bytesYesBinary in JSON, with padding
base64url4 chars per 3 bytesYesURLs and JWT segments
utf16le2 or 4NoWindows interop

Practical Edge Cases and Deep Dives

1. byteLength is not length

js
const s = 'café'; s.length; // 4 Buffer.byteLength(s, 'utf8'); // 5 Buffer.from(s).length; // 5

Any Content-Length, file offset, or chunk-splitting arithmetic on a string must use Buffer.byteLength, or a request body with an accented character is one byte short and the receiving parser hangs waiting for the rest. The same mistake in reverse — buf.length used as a character count — truncates a slice in the middle of a multi-byte sequence, producing U+FFFD at the boundary.

2. Splitting a UTF-8 stream corrupts text

js
// Wrong: a multi-byte character can straddle the chunk boundary socket.on('data', (chunk) => process.stdout.write(chunk.toString())); // Right const dec = new StringDecoder('utf8'); socket.on('data', (chunk) => process.stdout.write(dec.write(chunk))); socket.on('end', () => process.stdout.write(dec.end()));

A 3-byte character split across two 64 KiB reads becomes two invalid sequences, and toString() on each half yields replacement characters. StringDecoder holds the incomplete tail until the next chunk. On the encoding side the same rule applies: never assume a chunk ends on a character boundary, and never split a buffer at an arbitrary index when the content is text.

3. allocUnsafe and the data that was already there

js
const buf = Buffer.allocUnsafe(1024); // may contain bytes from a previous allocation

allocUnsafe is faster because it skips zeroing, and it is how pooled small buffers are carved out; the bytes it hands back can be remnants of something else the process handled — a decrypted payload, a token, part of another user's file. It is safe only when every byte is written before the buffer is read. For anything that is serialised, hashed, or sent over the wire, Buffer.alloc is the honest default, and the measurable cost is usually a rounding error next to the I/O that follows.

4. Transferring to a worker: the pool trap

js
import { Worker } from 'node:worker_threads'; const payload = Buffer.from('important data'); // may live in the 8 KiB pool const copy = new Uint8Array(payload); // own ArrayBuffer const worker = new Worker('./w.js', { workerData: copy, transferList: [copy.buffer], });

Transferring payload.buffer directly detaches the whole pool — including bytes of other buffers allocated nearby — and sends them to the worker. The bytes are not read by your code, but they were sent, which is a disclosure bug when one buffer holds a session token and another holds public data. new Uint8Array(payload) copies into a dedicated ArrayBuffer, and Buffer.from(payload) gives you a copy too. Also remember that after a transfer the source buffer is unusable (byteLength 0), and structuredClone(buf) returns a Uint8Array, not a Buffer.

5. Base64, padding, and URL safety

js
Buffer.from('user:pass').toString('base64'); // 'dXNlcjpwYXNz' Buffer.from(token, 'base64url'); // decode, tolerant of missing padding Buffer.from(emoji).toString('base64url'); // no '+', '/', or '=' to escape

Standard base64 uses + and /, which need URL encoding, and = padding, which some routers and signature schemes reject. base64url (Node 15.7+) implements RFC 4648 §5 and is what JWTs, signed URLs, and Authorization headers should use. Note that decoding a base64 string that was truncated is not an error — Node decodes what it can — so a token validation path must compare decoded bytes with timingSafeEqual, not strings.

6. Reading binary structure without a library

js
function parseHeader(buf) { if (buf.length < 16) throw new Error('short header'); const magic = buf.toString('latin1', 0, 4); const version = buf.readUInt16BE(4); const flags = buf.readUInt8(6); const size = buf.readBigUInt64BE(8); return { magic, version, flags, size }; }

read*/write* variants exist for UInt8, UInt16, UInt32, BigUInt64, Int*, Float, Double, in both LE and BE, with an offset that is bounds-checked (an out-of-range read throws ERR_OUT_OF_RANGE). Use latin1 for fixed-width magic strings so each byte maps to one character, and remember that readUInt32BE returns a Number that loses precision above 2^53 — use readBigUInt64BE for file sizes and identifiers from untrusted input.

Common Node.js Runtime Mistakes

SymptomCauseFix
Content-Length off by a few bytesstr.length used instead of Buffer.byteLengthCount bytes
`` characters in streamed textChunk split inside a multi-byte characterStringDecoder
Decoded token is null where a binary value was expectedWrong encoding, e.g. hex on base64 dataCheck the source encoding
Worker received unrelated dataTransferred a pooled ArrayBufferCopy with new Uint8Array(buf)
TypeError: Buffer.from(number)Deprecated numeric constructor removedBuffer.alloc(n)
ERR_OUT_OF_RANGE on readUInt32BEOffset past the end of the bufferValidate buf.length first
ERR_INVALID_ARG_TYPE with a stringPassing text where a Buffer is requiredBuffer.from(str)
Base64 with + breaks a URL signatureStandard base64 used in a URLbase64url
Sensitive data in a heap dumpallocUnsafe used and not fully writtenBuffer.alloc, buf.fill(0) after use
slice mutated the originalsubarray shares memoryCopy when the original must stay intact

Try matching browser tools

All tools

Small utilities that fit Node.js work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Node.js guides

More in Node.js

All Node.js guides

The guides filed next to this one in Node.js, in library order.