Parsing JSON and YAML in Shell Pipelines with jq and yq
jq is a filter language, not a pretty-printer. The operators that do the work, --arg versus string interpolation, in-place YAML edits, and how to tell which yq is installed before writing a command.

Quick Summary
jq is a filter language for JSON, not a formatter with flags: the program is a pipeline of filters, and every expression is evaluated against a value. jq '.items[] | select(.status == "open") | .id' reads as "iterate items, keep the open ones, take id". The two flags that change output in ways scripts depend on are -r (raw strings, no JSON quotes) and -c (one object per line), and the two that change input are -s (slurp the whole stream into one array) and -n (no input; useful with --arg).
yq exists in two incompatible families. Mike Farah's Go yq (v4) is a YAML-native editor with its own expression syntax and in-place editing (yq -i '.spec.replicas = 3' file.yaml). The Python yq (kislyuk) is a thin wrapper around jq: yq -y emits YAML, and everything else is jq syntax. Knowing which one is installed matters before writing a single command, because .a.b works in both while -i and -o do not.
The reliable shell pattern is to convert once and filter with the tool that matches your language: jq for API responses and JSON logs, Go yq for editing YAML in place (Kubernetes manifests, CI configs), and yq -o=json | jq when both are involved. This guide covers the operators that do the work (select, map, group_by, to_entries, //, try), quoting and --arg versus string interpolation, in-place YAML edits with multi-document files, and the error messages that mean "your input is not what you think it is".
Basic Syntax & Core Signatures
# jq: identity, field access, iteration
jq '.' response.json
jq -r '.data[0].name' response.json
jq -c '.[] | {id, name: .user.name}' response.json
# Conditions and reshaping
jq '[.[] | select(.enabled and (.score > 80))] | length' data.json
jq 'map({(.id|tostring): .name}) | add' data.json # build a map
jq 'group_by(.team) | map({team: .[0].team, n: length})' data.json
jq 'sort_by(-.created_at) | .[0:3]' data.json # newest three
jq 'to_entries | map(select(.value.active)) | from_entries' data.json
jq 'reduce .[] as $x (0; . + $x.amount)' data.json
# Defaults, optional access, errors
jq '.config.timeout // 30' data.json
jq '.items[]?.name' data.json # ?: no error when items is not iterable
jq 'try .a.b catch "missing"' data.json
# Input handling
jq -s 'add' parts.json # slurp array of documents, then add
jq -n --arg host db1 --argjson port 5432 '{host: $host, port: $port}'
jq -n --slurpfile cfg c.json '$cfg[0].name'
jq --rawfile sql query.sql -n '{q: $sql}'
jq -e '.error == null' resp.json && echo ok # exit status: 4 = false/null# Shell-safe quoting: single quotes for the program, --arg for data
jq -r --arg user "$USER" '.[] | select(.owner == $user) | .path' files.json
# Bad: the shell expands $USER and jq sees a bare word
jq -r ".[] | select(.owner == \"$USER\") | .path" files.json # accidental injection
# Formatted output
jq -r '.[] | [.id, .name, .score] | @tsv' data.json
jq -r '.[] | [.id, .name] | @csv' data.json
jq -r '.payload | @base64d' data.json# Go yq (mikefarah) v4
yq '.' config.yaml
yq '.spec.replicas' deploy.yaml
yq -i '.spec.replicas = 3' deploy.yaml
yq -i '.spec.template.spec.containers[0].image = "app:1.4.2"' deploy.yaml
yq -i 'del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' deploy.yaml
yq '.items[] | select(.kind == "Deployment") | .metadata.name' manifests.yaml
yq -o=json '.' config.yaml | jq '.database.host'
yq -N -o=props '.' config.yaml # key=value form
yq ea '. as $item ireduce ({}; . * $item)' a.yaml b.yaml # deep merge
# Python yq (kislyuk): jq syntax, -y for YAML output
yq -y '.services.web.image' docker-compose.ymlKey Comparison Table
| Task | jq | Go yq v4 |
|---|---|---|
| Read a path | .a.b[0].c | .a.b[0].c |
| Default value | // "x" | // "x" |
| Select | .[] | select(.x > 1) | .items[] | select(.x > 1) |
| Edit in place | Not available | -i |
| Output format | JSON only | -o json/yaml/props/xml/csv |
| Multi-document YAML | N/A | ea, explode, -s |
| Preserve comments | N/A | Yes in v4 (best effort) |
| Sort keys | -S | -P for pretty, --sort-keys |
| Compact output | -c | -o=json -I0 |
| Program from file | -f file.jq | --from-file |
| Flag | jq behaviour |
|---|---|
-r | Print strings without quotes (still errors on objects) |
-c | One compact JSON value per line, ideal for while read |
-e | Exit 1 if the result is false or null, 5 on errors: usable in if |
-s | All inputs into one array |
-n | No input; use input/$ARGS |
--arg | String variable (always quoted, no reparsing) |
--argjson | Variable parsed as JSON (numbers, objects) |
--tab / --indent n | Pretty-print controls |
Practical Edge Cases & Deep Dives
1. Cannot index ... with ...: the input shape is not what you think
$ jq '.name' < <(curl -s "$API")
jq: error (at <stdin>:0): Cannot index array with string "name"The response is an array; either index it (.[0].name) or iterate (.[] | .name). The same message appears when the body is an error page, in which case jq reports Cannot index string with string "name". Always check the shape before writing the filter:
curl -s "$API" | jq -e 'type' # "object", "array", "string"
curl -s "$API" | jq 'if type == "array" then .[0] else . end | keys'An empty input produces nothing at all rather than an error, which then silently becomes an empty variable:
name=$(curl -s "$API" | jq -r '.name')
[ -n "$name" ] || { echo "no name in response" >&2; exit 1; }2. --arg versus interpolation, and the two quoting bugs
# Correct: the value never passes through the jq parser
jq -n --arg path "$HOME/My Files" '{path: $path}'
# {"path": "/home/user/My Files"}
# Wrong: the shell substitutes, quotes break, and a quote in the value can inject jq syntax
jq -n '{path: "'"$HOME"'"}'The same distinction applies to building filters: never concatenate user input into a program string. --argjson is the correct way to pass numbers, arrays, or objects, since --arg 10 produces the string "10", which then fails a numeric comparison. If a whole filter must come from a variable, pass it with -f/--from-file rather than jq "$filter".
3. Building JSON, CSV, and shell-safe lines from a stream
# One JSON object per line: the format that works with while read and log pipelines
jq -c '.items[] | {id, name}' data.json
# TSV with explicit null handling, no header
jq -r '.items[] | [(.id|tostring), (.name // "-"), (.score|tostring)] | @tsv' data.json
# CSV with a header row
{ echo "id,name,score"
jq -r '.items[] | [.id, .name, .score] | @csv' data.json
} > report.csv
# Shell-safe assignment from JSON, one variable at a time
while IFS= read -r line; do
id=$(jq -r '.id' <<< "$line")
printf '%s\n' "$id"
done < <(jq -c '.items[]' data.json)@tsv and @csv escape separators and quote fields themselves, which is the only reliable way to produce tabular output from arbitrary strings. -r alone does not escape anything; a name containing a tab or a newline breaks the downstream parser.
4. Editing YAML in place without destroying it
# Go yq: quoting matters for keys with dots or slashes
yq -i '.metadata.labels."app.kubernetes.io/name" = "api"' deploy.yaml
# Append to an array, remove an element
yq -i '.spec.template.spec.containers[0].env += [{"name":"LOG_LEVEL","value":"info"}]' deploy.yaml
yq -i '.spec.template.spec.containers[0].env |= map(select(.name != "DEBUG"))' deploy.yaml
# Multi-document files: edit only the Deployment
yq -i 'select(.kind == "Deployment").spec.replicas = 3' manifests.yaml
# Deep merge a values file into a chart file (the Helm-ish pattern)
yq ea '. as $item ireduce ({}; . * $item)' values.yaml override.yaml
# Convert for jq, then back
yq -o=json '.' manifests.yaml | jq -c 'select(.kind=="Service")' | yq -P -o=yamlGo yq rewrites the document through its own parser, so comments survive in v4 but key order and blank lines can change, and anchors are expanded. On files under review, dry run with yq '.' file.yaml | diff -u file.yaml - first, and keep the file in version control rather than editing generated manifests in place.
5. Aggregation: group_by, reduce, and the // trap
jq '[.[] | {team, cost}] | group_by(.team) | map({team: .[0].team, total: (map(.cost) | add)})' data.json
# add on an empty array returns null, not 0 — the source of "null" totals
jq '[.[] | select(.ok)] | (map(.cost) | add) // 0' data.json
# Count by key without group_by
jq 'reduce .[] as $x ({}; .[$x.status] += 1)' data.json// is the alternative operator and it fires on false as well as null, so .flag // true turns an explicit false into true. When a boolean must survive, use if .flag == null then true else .flag end. Aggregations on empty input are the other silent failure: add of [] is null, length of null is 0, and a missing field propagates null through arithmetic, so validate at the boundary of the pipeline.
6. Which yq is installed
yq --version
# yq (https://github.com/mikefarah/yq/) version v4.44.1 -> Go yq
# yq 3.4.3 -> Python yq (jq wrapper)The two disagree on nearly every editing command: Python yq has -y for YAML output and no -i (you redirect), while Go yq uses -o, -i, and e (evaluate). A script that works on a developer's machine and fails in CI is usually this mismatch, and pinning the tool in the image is the fix:
command -v yq >/dev/null && yq --version | grep -q mikefarah || { echo "Go yq required" >&2; exit 1; }Common Runtime & Configuration Mistakes
| Symptom | Cause | Fix |
|---|---|---|
Cannot index array with string | Filter assumes an object | .[] / .[0], check type first |
| Output has quotes around strings | Missing -r | jq -r |
Value arrives as "10" | --arg makes a string | --argjson |
| Empty output, exit 0 | No input, or the filter matched nothing | -e, -n, or an explicit length check |
null where a number was expected | Missing field, or add of an empty array | // 0, has(), validation |
jq: error: Cannot iterate over null | .[] on null | .[]?, or // [] first |
| Filter broke with a quote in the value | String interpolation into the program | --arg |
false became true | // alternative on a boolean | if ... then ... else ... end |
| In-place edit changed key order or comments | YAML is rewritten by the parser | Review the diff, keep files in VCS |
yq: command not found inside a container | Neither yq in the image | Add the real binary, pin the version |
yq -i does nothing on Python yq | Different implementation | Redirect the output instead |
| Multi-document edit only touched the first | No ea/select | yq ea or a select per document |
Both tools are only as reliable as the shape they are given: check type and length at the top of a pipeline, pass data through --arg/--argjson, and write output with -r -c (jq) or an explicit -o (yq) so the next program in the pipeline sees one predictable value per line rather than a pretty-printed document.
Try matching browser tools
All toolsSmall utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Linux guides
All Linux guidesBash Indexed Arrays, Associative Maps, and Parameter Expansion
Quoting decides whether an array stays a list, and the expansion operators replace most external text tools. Includes set -u, mapfile, namerefs, and the bash 3.2 differences that break scripts on macOS.
Linux Memory, Swap, the OOM Killer, vmstat, and sysctl Tuning
MemAvailable versus used, the vmstat columns that prove thrashing, how the OOM killer picks its victim, and the sysctl keys that belong in /etc/sysctl.d instead of a one-off command.
Linux Firewalls: ufw, iptables, and nftables Rules and Persistence
First-match ordering, stateful filtering, and persistence: the three ideas behind every Linux firewall. Plus why Docker-published ports ignore ufw and how DOCKER-USER closes them.
Linux Users, Groups, sudoers, and PAM Access Control
The four account files, the sudoers grammar with aliases and drop-ins, and the PAM stack where logins actually succeed or fail. Includes the flags that lock an account without disabling SSH keys.
More in Linux
All Linux guidesThe guides filed next to this one in Linux, in library order.