Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Parsing JSON and YAML in Shell Pipelines with jq and yq

jq is a filter language, not a pretty-printer. The operators that do the work, --arg versus string interpolation, in-place YAML edits, and how to tell which yq is installed before writing a command.

~4 min readLinux
#Linux#CLI#JSON#Automation
3D illustration of JSON and YAML documents flowing through a filter funnel into clean output blocks

Quick Summary

jq is a filter language for JSON, not a formatter with flags: the program is a pipeline of filters, and every expression is evaluated against a value. jq '.items[] | select(.status == "open") | .id' reads as "iterate items, keep the open ones, take id". The two flags that change output in ways scripts depend on are -r (raw strings, no JSON quotes) and -c (one object per line), and the two that change input are -s (slurp the whole stream into one array) and -n (no input; useful with --arg).

yq exists in two incompatible families. Mike Farah's Go yq (v4) is a YAML-native editor with its own expression syntax and in-place editing (yq -i '.spec.replicas = 3' file.yaml). The Python yq (kislyuk) is a thin wrapper around jq: yq -y emits YAML, and everything else is jq syntax. Knowing which one is installed matters before writing a single command, because .a.b works in both while -i and -o do not.

The reliable shell pattern is to convert once and filter with the tool that matches your language: jq for API responses and JSON logs, Go yq for editing YAML in place (Kubernetes manifests, CI configs), and yq -o=json | jq when both are involved. This guide covers the operators that do the work (select, map, group_by, to_entries, //, try), quoting and --arg versus string interpolation, in-place YAML edits with multi-document files, and the error messages that mean "your input is not what you think it is".

Basic Syntax & Core Signatures

bash
# jq: identity, field access, iteration jq '.' response.json jq -r '.data[0].name' response.json jq -c '.[] | {id, name: .user.name}' response.json # Conditions and reshaping jq '[.[] | select(.enabled and (.score > 80))] | length' data.json jq 'map({(.id|tostring): .name}) | add' data.json # build a map jq 'group_by(.team) | map({team: .[0].team, n: length})' data.json jq 'sort_by(-.created_at) | .[0:3]' data.json # newest three jq 'to_entries | map(select(.value.active)) | from_entries' data.json jq 'reduce .[] as $x (0; . + $x.amount)' data.json # Defaults, optional access, errors jq '.config.timeout // 30' data.json jq '.items[]?.name' data.json # ?: no error when items is not iterable jq 'try .a.b catch "missing"' data.json # Input handling jq -s 'add' parts.json # slurp array of documents, then add jq -n --arg host db1 --argjson port 5432 '{host: $host, port: $port}' jq -n --slurpfile cfg c.json '$cfg[0].name' jq --rawfile sql query.sql -n '{q: $sql}' jq -e '.error == null' resp.json && echo ok # exit status: 4 = false/null
bash
# Shell-safe quoting: single quotes for the program, --arg for data jq -r --arg user "$USER" '.[] | select(.owner == $user) | .path' files.json # Bad: the shell expands $USER and jq sees a bare word jq -r ".[] | select(.owner == \"$USER\") | .path" files.json # accidental injection # Formatted output jq -r '.[] | [.id, .name, .score] | @tsv' data.json jq -r '.[] | [.id, .name] | @csv' data.json jq -r '.payload | @base64d' data.json
bash
# Go yq (mikefarah) v4 yq '.' config.yaml yq '.spec.replicas' deploy.yaml yq -i '.spec.replicas = 3' deploy.yaml yq -i '.spec.template.spec.containers[0].image = "app:1.4.2"' deploy.yaml yq -i 'del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' deploy.yaml yq '.items[] | select(.kind == "Deployment") | .metadata.name' manifests.yaml yq -o=json '.' config.yaml | jq '.database.host' yq -N -o=props '.' config.yaml # key=value form yq ea '. as $item ireduce ({}; . * $item)' a.yaml b.yaml # deep merge # Python yq (kislyuk): jq syntax, -y for YAML output yq -y '.services.web.image' docker-compose.yml

Key Comparison Table

TaskjqGo yq v4
Read a path.a.b[0].c.a.b[0].c
Default value// "x"// "x"
Select.[] | select(.x > 1).items[] | select(.x > 1)
Edit in placeNot available-i
Output formatJSON only-o json/yaml/props/xml/csv
Multi-document YAMLN/Aea, explode, -s
Preserve commentsN/AYes in v4 (best effort)
Sort keys-S-P for pretty, --sort-keys
Compact output-c-o=json -I0
Program from file-f file.jq--from-file
Flagjq behaviour
-rPrint strings without quotes (still errors on objects)
-cOne compact JSON value per line, ideal for while read
-eExit 1 if the result is false or null, 5 on errors: usable in if
-sAll inputs into one array
-nNo input; use input/$ARGS
--argString variable (always quoted, no reparsing)
--argjsonVariable parsed as JSON (numbers, objects)
--tab / --indent nPretty-print controls

Practical Edge Cases & Deep Dives

1. Cannot index ... with ...: the input shape is not what you think

bash
$ jq '.name' < <(curl -s "$API") jq: error (at <stdin>:0): Cannot index array with string "name"

The response is an array; either index it (.[0].name) or iterate (.[] | .name). The same message appears when the body is an error page, in which case jq reports Cannot index string with string "name". Always check the shape before writing the filter:

bash
curl -s "$API" | jq -e 'type' # "object", "array", "string" curl -s "$API" | jq 'if type == "array" then .[0] else . end | keys'

An empty input produces nothing at all rather than an error, which then silently becomes an empty variable:

bash
name=$(curl -s "$API" | jq -r '.name') [ -n "$name" ] || { echo "no name in response" >&2; exit 1; }

2. --arg versus interpolation, and the two quoting bugs

bash
# Correct: the value never passes through the jq parser jq -n --arg path "$HOME/My Files" '{path: $path}' # {"path": "/home/user/My Files"} # Wrong: the shell substitutes, quotes break, and a quote in the value can inject jq syntax jq -n '{path: "'"$HOME"'"}'

The same distinction applies to building filters: never concatenate user input into a program string. --argjson is the correct way to pass numbers, arrays, or objects, since --arg 10 produces the string "10", which then fails a numeric comparison. If a whole filter must come from a variable, pass it with -f/--from-file rather than jq "$filter".

3. Building JSON, CSV, and shell-safe lines from a stream

bash
# One JSON object per line: the format that works with while read and log pipelines jq -c '.items[] | {id, name}' data.json # TSV with explicit null handling, no header jq -r '.items[] | [(.id|tostring), (.name // "-"), (.score|tostring)] | @tsv' data.json # CSV with a header row { echo "id,name,score" jq -r '.items[] | [.id, .name, .score] | @csv' data.json } > report.csv # Shell-safe assignment from JSON, one variable at a time while IFS= read -r line; do id=$(jq -r '.id' <<< "$line") printf '%s\n' "$id" done < <(jq -c '.items[]' data.json)

@tsv and @csv escape separators and quote fields themselves, which is the only reliable way to produce tabular output from arbitrary strings. -r alone does not escape anything; a name containing a tab or a newline breaks the downstream parser.

4. Editing YAML in place without destroying it

bash
# Go yq: quoting matters for keys with dots or slashes yq -i '.metadata.labels."app.kubernetes.io/name" = "api"' deploy.yaml # Append to an array, remove an element yq -i '.spec.template.spec.containers[0].env += [{"name":"LOG_LEVEL","value":"info"}]' deploy.yaml yq -i '.spec.template.spec.containers[0].env |= map(select(.name != "DEBUG"))' deploy.yaml # Multi-document files: edit only the Deployment yq -i 'select(.kind == "Deployment").spec.replicas = 3' manifests.yaml # Deep merge a values file into a chart file (the Helm-ish pattern) yq ea '. as $item ireduce ({}; . * $item)' values.yaml override.yaml # Convert for jq, then back yq -o=json '.' manifests.yaml | jq -c 'select(.kind=="Service")' | yq -P -o=yaml

Go yq rewrites the document through its own parser, so comments survive in v4 but key order and blank lines can change, and anchors are expanded. On files under review, dry run with yq '.' file.yaml | diff -u file.yaml - first, and keep the file in version control rather than editing generated manifests in place.

5. Aggregation: group_by, reduce, and the // trap

bash
jq '[.[] | {team, cost}] | group_by(.team) | map({team: .[0].team, total: (map(.cost) | add)})' data.json # add on an empty array returns null, not 0 — the source of "null" totals jq '[.[] | select(.ok)] | (map(.cost) | add) // 0' data.json # Count by key without group_by jq 'reduce .[] as $x ({}; .[$x.status] += 1)' data.json

// is the alternative operator and it fires on false as well as null, so .flag // true turns an explicit false into true. When a boolean must survive, use if .flag == null then true else .flag end. Aggregations on empty input are the other silent failure: add of [] is null, length of null is 0, and a missing field propagates null through arithmetic, so validate at the boundary of the pipeline.

6. Which yq is installed

bash
yq --version # yq (https://github.com/mikefarah/yq/) version v4.44.1 -> Go yq # yq 3.4.3 -> Python yq (jq wrapper)

The two disagree on nearly every editing command: Python yq has -y for YAML output and no -i (you redirect), while Go yq uses -o, -i, and e (evaluate). A script that works on a developer's machine and fails in CI is usually this mismatch, and pinning the tool in the image is the fix:

bash
command -v yq >/dev/null && yq --version | grep -q mikefarah || { echo "Go yq required" >&2; exit 1; }

Common Runtime & Configuration Mistakes

SymptomCauseFix
Cannot index array with stringFilter assumes an object.[] / .[0], check type first
Output has quotes around stringsMissing -rjq -r
Value arrives as "10"--arg makes a string--argjson
Empty output, exit 0No input, or the filter matched nothing-e, -n, or an explicit length check
null where a number was expectedMissing field, or add of an empty array// 0, has(), validation
jq: error: Cannot iterate over null.[] on null.[]?, or // [] first
Filter broke with a quote in the valueString interpolation into the program--arg
false became true// alternative on a booleanif ... then ... else ... end
In-place edit changed key order or commentsYAML is rewritten by the parserReview the diff, keep files in VCS
yq: command not found inside a containerNeither yq in the imageAdd the real binary, pin the version
yq -i does nothing on Python yqDifferent implementationRedirect the output instead
Multi-document edit only touched the firstNo ea/selectyq ea or a select per document

Both tools are only as reliable as the shape they are given: check type and length at the top of a pipeline, pass data through --arg/--argjson, and write output with -r -c (jq) or an explicit -o (yq) so the next program in the pipeline sees one predictable value per line rather than a pretty-printed document.

Try matching browser tools

All tools

Small utilities that fit Linux work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Linux guides

More in Linux

All Linux guides

The guides filed next to this one in Linux, in library order.