Skip to content
How-To

6 Tests for Parsing Concatenated JSON Objects in Python

·10 min read
6 Tests for Parsing Concatenated JSON Objects in Python - How-To guide from JSON Prettifier

Quick answer: For a bounded string that intentionally contains adjacent JSON objects, use Python's JSONDecoder to parse one object at a time. Advance to the returned end index, skip only JSON whitespace, and reject any unexpected suffix. Keep json.loads for endpoints that promise a single document.

Two valid objects can make one invalid JSON document. The examples below use Python 3.12.14 and a deliberate contract: accept a complete, size limited string containing one or more JSON objects. Copy the Python blocks into one file in order and run them together.

1. How do you confirm that Extra data comes from a second object?

Compare the same text with a single document decoder and a prefix decoder. The single document decoder should reject the text, while the prefix decoder should return the first object and the index where it ended. Inspect the suffix before deciding whether the producer actually sent a batch.

import json
from contextlib import contextmanager

@contextmanager
def rejects(error):
    try:
        yield
    except error:
        pass
    else:
        raise AssertionError(f"Expected {error.__name__}")

source = '{"job":1}{"job":2}'
with rejects(json.JSONDecodeError):
    json.loads(source)
decoder = json.JSONDecoder()
first, end = decoder.raw_decode(source)
assert first == {"job": 1}
assert source[end:] == '{"job":2}'

The Python raw_decode reference describes the returned value and index. A prefix parse is useful for diagnosis because it gives you a precise point at which to inspect the remaining text. It does not identify the producer's intent. The same failure could come from a duplicate response, a diagnostic line appended to a file, or an export intentionally written as a series of records.

Start with the contract. If an HTTP endpoint promises one object, treating a second object as valid can hide a bug and make the result depend on which object you keep. If an existing batch producer intentionally emits adjacent objects, the reader should consume all of them and report a failure when the rest cannot be parsed. See the Python JSON decoding policy guide for checks that belong at a single document boundary.

When you investigate a real export, take a small redacted sample that includes the last complete record and the beginning of what follows. Avoid deciding that the whole file is concatenated JSON simply because two objects at the start decode. A mixed file might have a footer or status marker that belongs to a different part of the format. Ask the producer to define that boundary explicitly before teaching a reader to discard it.

2. How do you advance across whitespace and nested braces?

After decoding each object, resume at the returned index and skip only the four JSON whitespace characters. Let the decoder recognize the end of the next object. Searching for a closing brace or splitting on line breaks can cut through quoted strings, nested objects, or a record formatted across several lines.

def read_objects(text, *, max_chars=4096, max_records=16):
    if len(text) > max_chars:
        raise ValueError("input too long")
    records = []
    i = 0
    while True:
        while i < len(text) and text[i] in " \t\r\n":
            i += 1
        if i == len(text):
            if not records:
                raise ValueError("expected at least one object")
            return records
        if text[i] != "{":
            raise ValueError(f"expected object at character {i}")
        if len(records) >= max_records:
            raise ValueError("too many records")
        value, i = decoder.raw_decode(text, i)
        records.append(value)

assert read_objects(' \r\n{"note":"}","data":{"n":1}}\t{}\n') == [
    {"note": "}", "data": {"n": 1}}, {},
]
with rejects(ValueError):
    read_objects('{}\u00a0{}')

The JSON whitespace set is defined in RFC 8259. The second fixture uses a nonbreaking space, which is easy to miss in a text editor. It is not a JSON whitespace character. A generic whitespace test could accept this input and mask an encoding or producer issue. The sample uses the positional cursor supported by the CPython decoder implementation so a decoder error still refers to the original string.

To inspect the structure after extracting an individual synthetic record, paste that record into the JSON Viewer tree. Expand the nested object and check that the brace in the note remains part of the string. The viewer is for examining one document; it does not determine whether the surrounding file obeys your batch protocol.

3. How do you keep a batch reader from guessing scalar boundaries?

Define the allowed top level record type before writing the loop. If your producer promises objects, require an opening brace at each record boundary and reject numbers, strings, arrays, and literals there. A decoder can recognize a number, but it cannot recover an omitted delimiter between two intended numeric records.

assert decoder.raw_decode("12") == (12, 2)
assert read_objects('{"n":1}{"n":2}') == [{"n": 1}, {"n": 2}]
for bad in ('12', '[]{}', '{}true', '{}42{}'):
    with rejects(ValueError):
        read_objects(bad)

The first assertion shows why an unrestricted loop cannot fix a framing mistake. The text containing the digits one and two has the same representation whether the sender intended the number twelve or forgot to separate two numbers. Object boundaries can be found using the JSON grammar because the opening and closing braces are part of the structure. That property says nothing about whether the resulting object has the fields your application expects.

If you can change the producer, consider a format with explicit record boundaries. Wikipedia's JSON streaming overview surveys several approaches, including line delimiters and record separators. The JSON Lines guide describes a format for one JSON value per line. The important choice is a boundary that the sender and receiver both agree to, including what to do if the last record is incomplete. Adjacent objects are useful when reading an existing format; there is no need to reproduce that limitation in a new protocol.

The opening brace check applies only to the record root. Nested arrays and scalar values are still allowed. It also leaves Python's standard decoder defaults in place: the sample does not reject repeated member names or nonfinite numeric constants. If your input contract forbids either, configure and test the relevant decoder hooks separately. A reader can enforce its framing rule perfectly and still accept a value your business contract rejects. Keep separate fixtures for the two decisions.

4. How do you reject a corrupt tail after a valid first object?

Check the entire string before returning the batch. After one successful decode, continue until you reach the end after permitted whitespace. Reject trailing text and incomplete objects instead of returning a successful prefix. Otherwise an importer can report success while silently dropping the final records it was expected to process.

for bad in ('{} trailing', '{}{"job":', '{}{}!'):
    with rejects(ValueError):
        read_objects(bad)

published = []
with rejects(ValueError):
    published.extend(read_objects('{"job":1}{"job":'))
assert published == []

The function builds a local list, so a failure prevents the caller from receiving any of the records. This is a parsing guarantee in the fixture, not a database transaction. If you first parse the batch and then save records one by one, a failed write can still leave partial changes. When a whole batch must succeed or fail together, use the appropriate transaction or staging mechanism for the destination as well.

There are valid reasons to process records independently, such as a log pipeline that must keep earlier records after a later one fails. In that case, make partial acceptance an explicit part of the protocol. Report which records were accepted and where processing stopped. Without a delimiter that reliably marks the next record, skipping to the next brace after an error can resume inside a string or nested object. Recovery requires stronger information than the fact that a brace appears later.

Test the failure at the consumer boundary too. A caller may catch the exception and accidentally mark the job complete, even though the parser never returned a batch. The useful integration assertion checks both that no records were written and that the failed job remains visible for investigation. When retries are possible, decide whether a repeated copy of a valid batch is allowed or needs a stable identifier so that it cannot be applied twice.

5. How do you report the correct position after Unicode text?

Treat the decoder's position as an index into the decoded Python string. If an operator needs a UTF-8 byte position, convert the prefix up to that index into bytes under the same encoding assumptions. Test with a literal non-ASCII character because an ASCII only fixture cannot expose the difference.

sample = '{"city":"é"}{"n":}'
try:
    read_objects(sample)
except json.JSONDecodeError as exc:
    assert exc.doc is sample
    assert sample[exc.pos] == "}"
    assert len(sample[:exc.pos].encode("utf-8")) == exc.pos + 1
else:
    raise AssertionError("expected a decode error")

The Python JSONDecodeError reference documents the error position and line and column fields. In this fixture, the final brace is the point at which the decoder expected a value. The accented character occurs earlier, so the UTF-8 byte position is one larger than the string index. The conversion assumes the original bytes were UTF-8 and that no preprocessing changed the text. If a reader strips a byte order mark or normalizes newlines first, the original byte position needs separate accounting.

Choose an unambiguous label in error messages. A character index is suitable when the reader can highlight a position in the same decoded text; a byte offset is helpful for a tool that opens the raw file. Keep those units separate. If payloads can contain private information, a record number and a position are usually more useful in shared logs than an excerpt copied directly from the input.

For a human facing report, include the units and whether counting starts at zero or one. Python's position index is zero based, while its line and column numbers are one based. A user might paste the line and column into an editor but send a byte offset to a repair script. Verify the conversion on a fixture with a line break before the error if the report displays both forms. They should identify the same location in the same input.

6. How do you test limits without mistaking the reader for a streaming parser?

Set a limit on decoded text length and record count, then test values at and above both limits. These checks bound this sample reader's input and the number of records it returns. They do not limit the bytes already read from disk or the nesting of an individual JSON object.

assert read_objects('{}{}', max_chars=4, max_records=2) == [{}, {}]
with rejects(ValueError):
    read_objects('{}{} ', max_chars=4)
with rejects(ValueError):
    read_objects('{}{}{}', max_records=2)
with rejects(ValueError):
    read_objects(' \n ')

Apply a byte limit at the transport or file reading boundary before decoding untrusted input. By the time a function receives a string, memory has already been allocated for it. The character limit here is still useful for callers passing a complete string, but it cannot substitute for a bounded read. The count limit also has a different purpose from a nesting policy: one deeply nested object consumes only one record slot.

In a file migration, record the number of objects read and compare it to an independently known count when one is available. That helps catch a batch that ends cleanly but is missing records upstream. For ongoing traffic or inputs too large to hold in memory, choose a protocol and parser that can consume framed records incrementally. Adding a loop around raw_decode on a fully materialized string does not make reading incremental. Keep the simple reader for the bounded case it tests, and check the producer and consumer together before relying on it in a deployment.

For feeds that already use a record separator, the JSON text sequence tests cover that framing contract separately.

Frequently Asked Questions

Why does json.loads raise Extra data for two JSON objects?
The function expects one JSON document. After parsing the first object and skipping any following JSON whitespace, it raises an error if text remains. If the producer intentionally sends multiple objects, agree on a framing format or use a decoder that explicitly walks each object and checks the entire input.
Does raw_decode return a byte offset?
No. When raw_decode processes a Python string, the returned index belongs to that string. Characters outside ASCII can occupy more than one byte in UTF-8, so the index may differ from a byte position. Convert it deliberately if you need to report a location in the original encoded input.
Can raw_decode parse incomplete chunks from a socket?
It can parse a complete value present in a string, but it does not tell you whether a syntax error means the value is incomplete or permanently invalid. A socket reader needs a framing rule, limits on buffered data, and a policy for handling an incomplete record when the connection closes.
Can I split concatenated objects wherever I see a closing brace?
No. A closing brace may appear inside a quoted string or end an inner object. A JSON decoder understands both cases and can identify where the outer value ends. Use a small fixture containing a quoted brace and a nested object to catch accidental delimiter splitting in your parser.
Should I use concatenated JSON or JSON Lines for a new export?
If you control both sides and each record fits on one line, JSON Lines gives readers an explicit newline boundary. An existing producer may already emit adjacent objects without separators, which calls for a different reader. Document the format and decide how consumers should handle malformed records before shipping the export.

Working with JSON right now?

Paste it into JSON Prettifier to format, validate, or minify it. Everything runs in your browser.

Open JSON Prettifier