Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Go Strings, Runes, and Bytes: UTF-8 Encoding, Conversion, and Memory

Bytes versus runes, len that counts bytes, slicing that splits characters, Builder for linear concatenation, and normalization the standard library omits.

~5 min readGo
#Go#Strings#UTF-8
3D illustration of an immutable string byte slice decomposing into variable-width UTF-8 rune characters

A Go string is an immutable sequence of bytes, not characters. len counts bytes, indexing returns a byte, and slicing cuts at byte offsets — while range decodes UTF-8 and hands back one rune at a time with its starting byte index. Every bug in string handling comes from mixing those two views: treating a Korean character as one byte, slicing a multi-byte rune in half, or counting len where the user counts characters.

Quick Summary

  • string is a read-only byte slice: len(s) is bytes, s[i] is a byte, and s[i:j] is a byte range.
  • range over a string decodes UTF-8, yielding byte offsets and rune values; the index jumps by 1 to 4.
  • []rune(s) decodes into a slice of code points and allocates; []byte(s) copies the bytes; string(b) copies back.
  • Invalid UTF-8 decodes as utf8.RuneError (U+FFFD) with size 1, so corrupted input turns into replacement characters rather than an error.
  • Concatenation with += in a loop copies the whole string each time; strings.Builder with Grow is the linear-time form.
  • Strings compare with == bytewise, which is exact and fast but not case- or normalization-insensitive; strings.EqualFold handles case.
  • Since Go 1.20, unsafe.String and unsafe.Slice convert without copying — useful in hot paths and dangerous the moment the bytes are mutated or pooled.
  • Unicode normalization is not in the standard library; canonically equivalent strings compare unequal.

Comparison of string representations and conversions

ExpressionDirectionAllocatesCostUse
[]byte(s)string → bytesyes, copiesO(n)writing, parsing
string(b)bytes → stringyes, copiesO(n)keys, output
[]rune(s)string → runesyes, decodesO(n)per-character logic
string(r)rune → stringyesO(1)building one character
len(s)byte lengthnoO(1)buffers, limits
utf8.RuneCountInString(s)rune countnoO(n)user-visible length
range sdecodenoO(n)iterating characters
unsafe.String(&b[0], len(b))bytes → string, no copynoO(1)hot path, read-only

Basic syntax and type signatures

go
type byte = uint8 type rune = int32 s := "안녕하세요" len(s) // 15 bytes: 5 characters × 3 bytes utf8.RuneCountInString(s) // 5 runes s[0] // 0xEC, the first byte of 안 string(s[0:3]) // "안" — the full first rune for i, r := range s { fmt.Printf("%d %c %U\n", i, r, r) // offsets 0,3,6,9,12 } for i := 0; i < len(s); i++ { fmt.Printf("%02x ", s[i]) // bytes, not characters } runes := []rune(s) // [안 녕 하 세 요] fmt.Println(len(runes)) // 5 fmt.Println(string(runes[0:2])) // "안녕"
go
// Byte vs rune literals. b := 'a' // byte (uint8) if it fits, but the literal type is rune r := '안' // rune, value 0xC548 fmt.Println(b, r, string(r))

The distinction between len and utf8.RuneCountInString is the single most important one for user-facing code: a 280-character limit enforced with len rejects a Korean or Japanese post at roughly a third of its actual length, and a database column sized in characters truncates at a different point than a Go check.

Practical edge cases and deep dives

1. Slicing bytes can cut a rune in half

go
s := "안녕하세요" half := s[:2] // cuts the first 3-byte rune fmt.Println(utf8.ValidString(half)) // false fmt.Println(half) // "\xec\x95" — invalid UTF-8 // The correct prefix by characters. runes := []rune(s) prefix := string(runes[:2]) // "안녕" // Or without allocating the whole slice, walking runes: func truncateRunes(s string, n int) string { count := 0 for i := range s { // i is the byte index of each rune start if count == n { return s[:i] } count++ } return s }

for i := range s yields each rune's starting byte offset, and the final offset is len(s), so this loop can cut at a rune boundary without decoding the string twice. A byte slice that ends mid-rune is not an error the compiler or fmt will report — it prints replacement characters, and json.Marshal of such a string produces \ufffd, which is how invalid UTF-8 reaches a client.

go
// Truncating a string for storage: check the boundary, not just the length. func clip(s string, maxBytes int) string { if len(s) <= maxBytes { return s } cut := maxBytes for cut > 0 && !utf8.RuneStart(s[cut]) { cut-- } return s[:cut] }

2. Counting and reversing by runes

go
func reverse(s string) string { r := []rune(s) for i, j := 0, len(r)-1; i < j; i, j = i+1, j-1 { r[i], r[j] = r[j], r[i] } return string(r) } fmt.Println(reverse("안녕")) // "녕안" fmt.Println(reverse("hello")) // "olleh" func reverseWrong(s string) string { b := []byte(s) for i, j := 0, len(b)-1; i < j; i, j = i+1, j-1 { b[i], b[j] = b[j], b[i] } return string(b) // invalid UTF-8 for anything non-ASCII }

Reversal is the clearest demonstration that bytes and characters are different views: reversing bytes reverses the bytes inside each character. The same reasoning applies to substring search by index (strings.Index returns a byte offset, which is correct to use for slicing), padding (fmt.Sprintf("%-10s", s) pads by display width in bytes for some formatters), and column alignment in terminals.

utf8.RuneCountInString is O(n) but allocation-free, and utf8.RuneLen(r) gives the byte length of one rune. For repeated counting of the same string, caching the count is usually better than calling either in a loop.

3. Building strings without quadratic copying

go
// Quadratic: each += allocates and copies the whole prefix. func joinBad(parts []string) string { out := "" for _, p := range parts { out += p + "," } return out } // Linear: Builder grows a buffer and converts once. func joinGood(parts []string) string { var b strings.Builder b.Grow(len(parts) * 8) // a hint, not a limit for _, p := range parts { b.WriteString(p) b.WriteByte(',') } return b.String() } // Equivalent for byte-oriented work: func joinBuffer(parts []string) string { var buf bytes.Buffer buf.Grow(len(parts) * 8) for _, p := range parts { buf.WriteString(p) } return buf.String() }

strings.Builder avoids the final copy that bytes.Buffer.String() performs: String() uses unsafe internally to hand back the buffer's bytes as a string, which is safe because the builder refuses further writes once String has been called and never exposes the bytes for mutation. Grow reduces reallocations but does not bound the result — passing a total that is too small is harmless. For a few short concatenations, + is fine; the benchmark difference appears around the first hundred parts.

4. Conversions, allocations, and the zero-copy escape hatch

go
// Every conversion allocates. In a hot loop that allocation dominates. cache := map[string][]byte{} func lookupBad(key []byte) ([]byte, bool) { v, ok := cache[string(key)] // allocates a string copy for the lookup return v, ok } // Keeping the key a string avoids the conversion at every call site. func lookupGood(key string) ([]byte, bool) { v, ok := cache[key] return v, ok } // Zero-copy, read-only view (Go 1.20+). func bytesToString(b []byte) string { if len(b) == 0 { return "" } return unsafe.String(unsafe.SliceData(b), len(b)) }

unsafe.String produces a string that shares the byte slice's memory. That is only valid while the bytes are never mutated and never returned to a pool; a sync.Pool buffer that is reused after the conversion leaves the string pointing at unrelated data, which is the kind of bug that shows up as garbled output minutes later. The safe default is a copy, and the safe optimization is to avoid the conversion entirely: keep keys as string, keep buffers as []byte, and convert at the boundary.

go
// Converting once per call instead of once per element. func parseAll(lines []string) [][]byte { out := make([][]byte, 0, len(lines)) for _, l := range lines { out = append(out, []byte(l)) // one allocation per line, unavoidable here } return out }

5. Case, normalization, and invalid input

go
fmt.Println(strings.EqualFold("Straße", "STRASSE")) // false: ß does not fold to SS fmt.Println(strings.EqualFold("İ", "i")) // false: Turkish dotted capital I fmt.Println(strings.ToUpper("straße")) // "STRAßE": simple mapping keeps ß // Invalid UTF-8 becomes U+FFFD on decode. bad := string([]byte{0xff, 0xfe, 'a'}) fmt.Println(utf8.ValidString(bad)) // false fmt.Printf("%q\n", bad) // "\xff\xfea" — %q escapes, plain output shows replacement glyphs for i, r := range bad { fmt.Println(i, r == utf8.RuneError) // true for the invalid bytes } // Cleaning and validating at a boundary: func sanitize(s string) (string, error) { if !utf8.ValidString(s) { return "", fmt.Errorf("invalid utf-8") } return strings.ToValidUTF8(s, ""), nil }

EqualFold implements Unicode simple case folding, which handles most scripts but not the full case mappings (ß to SS) or locale-sensitive rules (Turkish dotless i). Text that will be compared as user input — emails, usernames, search — needs strings.ToLower plus a documented rule, or the golang.org/x/text/cases package for locale-aware mapping. Normalization is a separate axis: "é" as one code point and as e plus a combining accent are different byte sequences that look identical, and the standard library does not normalize them; golang.org/x/text/unicode/norm does, and it is required before comparing user-supplied text that may come from macOS (NFD) and Windows or the web (NFC).

Common mistakes and compiler errors

Error or symptomCauseFix
A Korean string reports three times the expected lengthlen counts bytesutf8.RuneCountInString
Truncated text shows replacement glyphs (U+FFFD)sliced mid-runecut at a rune boundary
Reversed text is garbagereversed bytesreverse []rune
s[0] compares wronglybyte, not charactercompare []rune(s)[0] or decode
Slow string building+= in a loopstrings.Builder
strings.Builder panics after copythe builder was copied by valuepass a pointer, never copy
unsafe.String output changes laterthe backing bytes were reusedcopy, or never mutate the source
Two identical-looking strings compare unequaldifferent Unicode normalizationnormalize with x/text/unicode/norm
Case-insensitive match fails== after ToLower misses locale rulesstrings.EqualFold, or cases
index out of range slicing by character countbyte offsets used as rune indicesconvert to []rune first

The mental model that prevents nearly all of these: a string is bytes, a rune is a code point, and a grapheme cluster (what a user calls a character) may be several runes. len and indexing work on bytes, range works on runes, and anything that needs user-perceived characters needs a grapheme library. Choosing the right one at the point where a string enters the program — validating UTF-8 at the boundary, deciding whether limits are in bytes or characters, normalizing before comparison — is far cheaper than fixing the places that guessed wrong later.

Try matching browser tools

All tools

Small utilities that fit Go work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Go guides

More in Go

All Go guides

The guides filed next to this one in Go, in library order.