Quick answer: Check four things before using jq --stream on a large JSON file: which events represent values, which paths identify your data, how much state the filter retains, and what happens after a parse error. Test these on a tiny fixture before pointing the filter at the real export.
If a file contains one giant array, a compact output filter does not change how the input gets parsed. Streaming mode changes the input to the filter itself. The examples below were tested with jq 1.7 on local fixtures. The jq manual defines the behavior; Wikipedia's jq overview provides background on the language.
1. Which streaming events actually carry a value?
Check for a two-element event whose first element is a path array before treating the second element as data. Ordinary value events can carry scalars or empty containers. Path-only events mark container endings, so reading their missing second element as though it were an explicit null can produce a false match.
The streaming section of the jq manual describes the path representation. Object member names appear as strings and array offsets as numbers. When an event carries a value, those path components identify where the value was found. A path-only event has a different purpose; it is part of the information that makes reconstruction possible.
Use a fixture with real null, an empty array, and a nonempty nested object. The null should still appear after you filter out path-only events. The empty array should also appear because it is a leaf in this representation, even though it is a container in the original JSON. Testing only strings and numbers misses both cases.
printf '%s\n' '{"rows":[{"id":"a","note":null},{"id":"b","note":[]}],"meta":{}}' > fixture.json
jq -c --stream 'select(length == 2 and (.[0] | type == "array"))' fixture.json
The five value events correspond to the two identifiers, the null note, the empty note array, and the empty metadata object. The paths tell you which is which; the second element alone does not. Before adapting the fixture, paste the original small document into JSON Viewer to inspect its nesting. The viewer displays JSON structure, while the command above shows how jq traverses that structure.
Remember that the event shape depends on the mode. With error reporting enabled, a two-element array can represent a parse error instead of data. Its first element is an error string rather than a path array. The path-type check prevents that particular shape from being mistaken for a value, but merely skipping error events is not an acceptable failure policy for a complete export.
Before using event counts as row counts, decide which event stands for a row in your schema. A row with three scalar fields has more value events than a row with one, while an empty row can appear as a single empty-object event. Count a specific required field only if the contract actually makes it unique and present per row. Otherwise, test a row boundary or use complete records.
2. How do you select one field without matching lookalike paths?
Match the full path shape, including its depth, fixed member names, and the numeric position of the array index. Checking only the last component can collect unrelated fields from nested objects or metadata. Pair the path with a value-type check when the output contract requires a specific type, such as string identifiers.
Imagine the export has an identifier on every row, an identifier inside a nested owner object, and an identifier in top-level metadata. All three paths end with the same name, but only the direct row identifier belongs in the result. Add those lookalikes to the fixture so a too-broad filter fails visibly instead of accidentally passing a simple happy-path sample.
printf '%s\n' '{"rows":[{"id":"a","owner":{"id":"owner-a"}},{"id":"b"},{"id":null}],"meta":{"id":"export"}}' > ids.json
jq -c --stream '
select(length == 2 and (.[0] | type == "array"))
| select(.[0] | length == 3 and .[0] == "rows"
and (.[1] | type == "number") and .[2] == "id")
| .[1]
| select(type == "string")
' ids.json
The command produces two JSON strings, a and b, each quoted as JSON on its own line. It leaves out the nested owner, the export identifier, and the null. If null should instead trigger a contract failure, change the policy deliberately. A filter that skips a bad row is appropriate for some exploratory work, but it changes the meaning of a report that claims to account for every input row.
Keep the array index if a downstream consumer needs to identify the source position. Emitting the identifier alone loses that relationship. Also distinguish the path array used by jq from a JSONPath query expression: they are different representations. The jq cheat sheet covers ordinary field selection; the same visual field name in stream mode usually requires a path check.
Decide whether your downstream reader expects JSON values or raw strings. Compact JSON preserves quoting and escaping, so an identifier containing a newline can remain one JSON output record. Switching to raw output changes that framing. A fixture with an escaped newline is useful if the output will feed line-oriented tooling and identifiers may contain more than simple alphanumeric characters.
Add a fixture with a similarly named object where an array was expected. A path through an object member named zero contains the string zero, whereas an array offset uses a number. Testing the type of the component keeps those cases separate. That small negative control is especially helpful when two producer versions package similar values under different shapes and neither version is guaranteed to reach the job first.
3. How do you keep the summary from growing with the file?
Use a reducer whose state has a fixed shape, such as a count and a sum, and update it as qualifying events arrive. Stream parsing alone does not bound the memory used by your filter. Collecting all matches in an array or rebuilding an entire subtree can reintroduce substantial buffering.
This example uses null input mode so the program starts once, and then consumes the file through inputs. The reducer sees each event in turn. An explicit path check ensures that a number in metadata cannot change the totals. The state keeps only the two running fields; it never stores a list of matching rows.
printf '%s\n' '{"rows":[{"qty":2},{"qty":3},{"qty":null}],"meta":{"qty":100}}' > quantities.json
jq -cn --stream '
reduce inputs as $e ({count:0,sum:0};
if ($e | length == 2 and (.[0] | type == "array")
and (.[0] | length == 3 and .[0] == "rows"
and (.[1] | type == "number") and .[2] == "qty")
and (.[1] | type == "number"))
then .count += 1 | .sum += $e[1]
else .
end)
' quantities.json
The expected result is a count of two and a sum of five. The null does not contribute, and the metadata number is outside the selected path. A zero quantity, by contrast, would contribute to the count even though it would not increase the sum. That distinction is why the filter checks the type instead of relying on whether the number looks meaningful in a report.
A fixed number of accumulator fields is a property of this filter, not a benchmark or an absolute memory guarantee. The parser still needs to represent the current event, path, and value, and a large string can be expensive even without collecting records. Test realistic nesting and value sizes on the same jq version and resource limits you use in production.
The choice also depends on the source format. If you control the producer and each record is independent, JSON Lines can let you process complete records without converting their fields into events. If you receive one giant document and need only a few leaves, a path-based reducer can avoid reconstructing rows you do not need.
Choose the summary fields before writing the reducer. A count of qualifying leaves can be useful, but it is not a count of rejected rows, missing quantities, or all input records. If the report needs those distinctions, add explicit counters and construct a fixture where they differ. Do not infer that a missing event means null: a row without the member and a row with an explicit null contribute different evidence to an audit.
4. How do you avoid publishing a partial result after a parse failure?
Make parse errors an explicit failure condition and stage the output until the command succeeds. A streaming filter can emit valid results before encountering invalid input later in the file. Check a malformed fixture with a valid prefix, because tests that fail immediately cannot demonstrate what happens to output already written.
The jq manual documents stream-errors mode, where errors arrive as events. That means a filter that merely selects valid-looking events can discard the evidence of a problem and continue. The command below uses a separate branch to fail explicitly. It stages output in the destination directory so a successful rename can publish the completed file without exposing the intermediate contents through the final name.
printf '%s\n' '[1,]' > broken.json
tmp=$(mktemp ./stream-result.XXXXXX) || exit 1
if jq -cn --stream-errors '
inputs
| if (.[0] | type) == "string" then error(.[0])
elif length == 2 then .[1]
else empty
end
' broken.json > "$tmp"; then
mv -- "$tmp" result.jsonl
else
rm -f -- "$tmp"
printf '%s\n' 'Input rejected; no new result was published.' >&2
exit 1
fi
The malformed fixture produces an early value event and then a parse error. The failure branch discards the staged file. Repeat with a valid array to verify that the success branch can publish a complete result. A script that always discards output would pass the rejection test alone, so the accepted case is part of the evidence.
This small shell example focuses on parser failure. If another program supplies input through a pipeline, also verify how the calling shell reports upstream failures. If the result replaces an important file, handle publication errors and interruption cleanup in the surrounding job. A successful jq parse does not prove that the upstream source completed its work or that the destination is durable.
Finally, define what complete means for the input format. The error guard rejects reported syntax problems, but it does not enforce a particular root schema or require exactly one top-level document. If either matters, test that boundary separately. For syntax checks outside the processing job, see the large JSON validation guide. Keep the fixtures alongside the command so future changes to the jq version or filter can be checked before an incomplete export becomes a published result.
For a scheduled export, include a previously published result in the rejection test. The command should leave that file untouched when the new input is malformed. Test the positive case with a different expected output so the check detects a script that exits successfully but never replaces the old result. This verifies the job's observable contract, not just the parser's return code.