Skip to content
How-To

4 Python json.tool Checks for CI Pipelines

·9 min read
4 Python json.tool Checks for CI Pipelines - How-To guide from JSON Prettifier

Quick answer: In CI, check the formatter's exit status, decide which input policies need a separate validator, verify the output format, and publish only after success. Python's json.tool is useful for formatting and basic parsing, but a successful command cannot tell you whether the result meets your application contract.

A formatter can fail after a build has already created its output file. These examples exercise the command as a subprocess on Python 3.12.14. Put the blocks in one file in order, then run that file with the Python interpreter used by your job.

1. How do you make CI catch a failed json.tool command?

Check the formatter process's exit status before treating its output as an artifact. Test one valid document and one malformed document so you know the command ran in both cases. A file's presence, a printed diagnostic, or the status of an unrelated pipeline stage cannot substitute for that check.

import json
import subprocess
import sys

def run(text, *args):
    return subprocess.run(
        [sys.executable, "-m", "json.tool", *args],
        input=text, text=True, capture_output=True, check=False,
    )

good = run('{"ready":true}')
assert good.returncode == 0
assert json.loads(good.stdout) == {"ready": True}

bad = run('{"ready":}')
assert bad.returncode != 0
assert bad.stderr.strip()

The Python 3.12 command line documentation describes the formatter invocation. The test uses the interpreter running the harness, so a local pass refers to that exact runtime. In a real job, pin the runtime you intend to test and record its version where the build can report it. Otherwise a developer can reproduce a command with a different executable and see a different result.

Keep the positive control. A broken interpreter path or an unavailable module can also yield an unsuccessful command, and a test that only checks the malformed fixture could pass for the wrong reason. Do not match the full diagnostic text unless your application specifically promises it. The result status and the presence of a diagnostic establish enough for this example. The Python JSON module guide covers error handling when your program calls the library directly.

Shell pipelines need particular care. For example, a successful downstream command can make a pipeline appear successful in a shell that reports only the last stage. Decide where to check failure based on the shell and runner you use, or invoke the formatter directly and check its result before invoking any follow-up command. Use a failing fixture in the real workflow as well as the subprocess test; that confirms that the job reports failure instead of uploading an artifact.

Also decide what should happen when a job cannot start the interpreter at all. That is an execution failure, not a malformed document. An operator needs different next steps for a missing executable, unreadable input, and a parser diagnostic. Let the workflow fail in all three cases, but preserve enough context to identify the step that failed. In a reusable script, include the command name and exit status and avoid sending diagnostics to the output stream intended for JSON.

2. Which input rules should you test outside json.tool?

Test rules such as rejecting repeated names and nonfinite numbers against the original text, before formatting it. The default Python JSON decoder accepts some input that stricter consumers reject. Once a formatter has collapsed repeated names into one mapping entry, a later check of the output cannot recover the original duplicates.

duplicates = run('{"mode":"safe","mode":"fast"}')
assert duplicates.returncode == 0
assert json.loads(duplicates.stdout) == {"mode": "fast"}
assert duplicates.stdout.count('"mode"') == 1

nonfinite = run('{"reading":NaN}')
assert nonfinite.returncode == 0
assert "NaN" in nonfinite.stdout

The Python standards compliance notes explain both defaults. The RFC 8259 object and number rules recommend unique member names, and the RFC's number grammar does not allow NaN. A successful Python parse does not prove that another parser accepts the same text or interprets repeated names the same way. For background on the format, see Wikipedia's JSON overview.

If your product rejects repeats, a decoder that exposes the original member pairs can detect them while there is still evidence. If your contract rejects nonfinite values, configure a parser policy for those constants. Keep these rules separate from a schema check, which generally operates on an already decoded instance. The Python decoder policy tests show how to test those choices.

For visual inspection, paste a small synthetic formatted document into the JSON Viewer tree and check which fields survived. A viewer can help confirm the shape that a formatter emitted. It cannot recover repeated names already discarded by the formatter, and it cannot tell you whether your target system accepts the document. Compare the behavior of the actual consumers when compatibility matters.

Avoid using formatted output as the only evidence in a failed build. If the fixture is safe to retain, keep the original and the candidate as distinct files. For real payloads, apply your normal rules about retaining sensitive data; a diagnostic naming the rejected field or policy can often explain the failure without preserving the full input.

Be deliberate about which stage is allowed to normalize the file. Sorting keys and changing indentation are presentation choices; losing a repeated member is a data decision. If a pipeline formats before enforcing policy, it can approve an output that its own input rules should have rejected. Test the validation order using a fixture whose formatted form would otherwise pass. That makes the check sensitive to someone rearranging the steps later.

3. How do you verify that formatted JSON Lines still has one record per line?

Run the formatter with JSON Lines input mode and compact output, then parse each output line independently and compare the complete sequence of values. JSON Lines input mode tells the tool how to read the source. Without an output formatting choice, pretty printing can split a single record across multiple lines.

batch = '{"job":1}\n{"job":2}\n'
pretty = run(batch, "--json-lines")
assert pretty.returncode == 0
assert len(pretty.stdout.splitlines()) > 2

compact = run(batch, "--json-lines", "--compact")
assert compact.returncode == 0
assert [json.loads(line) for line in compact.stdout.splitlines()] == [
    {"job": 1}, {"job": 2},
]

partial = run('{"job":1}\n{"job":}\n', "--json-lines", "--compact")
assert partial.returncode != 0
assert json.loads(partial.stdout.strip()) == {"job": 1}

The Python CLI options describe the input and output switches. The final fixture also shows why status matters for a line based stream. The tool emitted one valid record before encountering the malformed line. A consumer that checks only whether the first record parses can mistake that prefix for a complete export. In a release job, compare the record count and contents when you have an independent expectation, and discard staged output when the formatter fails.

Use a fixture with at least two records of different contents. A one line sample cannot catch a loop that stops after the first record or produces the same object twice. If record order matters, assert the exact sequence instead of comparing a set. If the producer permits blank lines or a final incomplete record, make those rules explicit and test them separately; this example expects both lines to hold complete JSON values. See the JSON Lines format guide for the distinction between a document and a sequence of line framed records.

This harness uses standard input. A CPython issue about JSON Lines and a named input file is another reason to run tests with the same invocation your job uses. The example does not establish the behavior of named input on all Python versions. Check the flags and file mode that your workflow actually passes.

A record count alone is also incomplete. For instance, two records might survive but arrive in reverse order, or one might be replaced with a copy of the other. The assertion compares parsed values to catch those changes without coupling the test to spaces or a final newline. If a downstream protocol has a byte level requirement, such as a mandatory final newline, add a separate assertion for it. A value comparison cannot verify details that the decoder ignores.

4. How do you protect the published file when formatting fails?

Write the candidate to a separate file beside the destination, check the formatter status, and replace the destination only after the candidate passes your checks. A failed process can leave partial output. Staging keeps that output away from readers and lets the previous successful file remain available during a failed attempt.

from pathlib import Path
from tempfile import TemporaryDirectory

def publish(text, destination):
    with TemporaryDirectory(dir=destination.parent) as directory:
        candidate = Path(directory) / "candidate.json"
        with candidate.open("w", encoding="utf-8") as handle:
            subprocess.run(
                [sys.executable, "-m", "json.tool"],
                input=text, text=True, stdout=handle, stderr=subprocess.PIPE,
                check=True,
            )
        candidate.replace(destination)

with TemporaryDirectory() as directory:
    dest = Path(directory) / "export.json"
    publish('{"version":1}', dest)
    try:
        publish('{"version":}', dest)
    except subprocess.CalledProcessError:
        pass
    else:
        raise AssertionError("expected formatting failure")
    assert json.loads(dest.read_text()) == {"version": 1}
    publish('{"version":2}', dest)
    assert json.loads(dest.read_text()) == {"version": 2}

The test uses a temporary directory beneath the destination directory so the staging and destination paths are on the same filesystem in this fixture. It closes the file before attempting the replacement. The Python path replacement reference describes the operation. This example checks that a failed formatter leaves the previous output intact and a successful attempt replaces it. It does not establish crash durability, coordinate concurrent writers, or preserve custom file permissions.

Put any required schema or content check before the replacement. A formatter can exit successfully while producing a document the application rejects, such as an object with a missing identifier. In that case the replacement should still be withheld. Extend the fixture with a syntactically valid but contract invalid document if your job performs that check. Then test that both failure paths preserve the earlier artifact and give operators distinct diagnostics.

The same principle applies to CI artifact upload. Keep the publish step dependent on successful formatting and validation, and point it at the final path rather than a directory full of candidates. Avoid leaving old artifacts where a later stage might upload them as if they were current. For a local file that must retain its last valid contents, the fixture tests exactly that promise. A build job can choose to fail outright instead; what matters is that the outcome is explicit and tested.

Run the publication fixture against a fresh destination as well if your workflow creates a file on its first successful run. Decide what a failed first run leaves behind: in this pattern the final path should still be absent. For a later failure the same path should contain the previous version. Those two starting conditions expose different mistakes, such as creating the final file before the command starts or deleting the previous file as part of cleanup. Keep the fixture on a disposable directory so both outcomes can be observed without touching a real export.

Frequently Asked Questions

Does python -m json.tool return a nonzero exit status on invalid JSON?
A syntax error in a document causes the tested command to exit unsuccessfully. A CI step should check the status of the actual formatter process before consuming its output. If a pipeline runs another command afterward, verify that your shell propagates the formatter failure or explicitly check each stage.
Does json.tool reject duplicate object keys?
The default Python JSON decoder accepts repeated object names and retains the last value for each name. Formatting can therefore remove evidence that the input contained duplicates. To reject repeats, inspect member pairs with a custom decoder before the formatter rewrites the document as an ordinary mapping.
Can json.tool format JSON Lines without breaking record boundaries?
Use the JSON Lines input option with a compact or unindented output option and verify that each output line parses independently. Default pretty printing may spread an object across several physical lines. The input option alone does not guarantee that the resulting file still has one record per line.
Should I use the same path as input and output?
Use a separate staging file and replace the destination only after the command and any required checks succeed. A direct shell redirect can truncate a file before the formatter reads it. Staging also keeps a failed formatting attempt from replacing the previous successful output.
Does json.tool validate a JSON Schema?
No. It decodes and formats JSON using Python's standard library behavior. A successful run does not prove that the document has required fields, satisfies a schema, or follows a stricter duplicate name or number policy. Run the contract checks separately before publishing the result.

Working with JSON right now?

Paste it into JSON Prettifier to format, validate, or minify it. Everything runs in your browser.

Open JSON Prettifier