Quick answer: Test JSON Schema error reporting at six boundaries: failure classification, instance pointers, missing properties, branch context, stable bounded output, and safe public diagnostics. These checks use Draft 2020-12 and python-jsonschema examples. The reporting adapter needs its own fixtures because rejecting bad input does not guarantee a useful or safe error response.
A request fails correctly, but the form highlights the wrong field. Another error response includes the rejected secret in its message. Both applications have working validation and a broken reporting boundary.
The API error-response guide covers the broader HTTP response shape. This article focuses on translating schema failures into that shape. Wikipedia’s data-validation overview supplies background; library documentation and the linked standards define the specific behavior tested here.
1. Does the adapter distinguish malformed JSON, an invalid schema, and invalid data?
Test parsing, schema checking, and instance validation as separate stages. Malformed JSON has no parsed instance to validate, an invalid schema is a service configuration problem, and invalid data violates an otherwise usable contract. Keep these failure categories distinct so a deployment defect is not reported as a customer field error or silently converted into acceptance.
import json
from jsonschema import Draft202012Validator
from jsonschema.exceptions import SchemaError
try:
json.loads('{"count":')
except json.JSONDecodeError:
pass
else:
raise AssertionError("Malformed fixture unexpectedly parsed")
try:
Draft202012Validator.check_schema({"type": "not-a-json-type"})
except SchemaError:
pass
else:
raise AssertionError("Invalid schema unexpectedly accepted")
For the third category, a valid integer schema should reject the string "three". That failure belongs to the instance, unlike either example above. The validator interface documentation distinguishes schema errors from instance validation errors; your adapter should preserve that distinction even if the public response uses fewer internal details.
Check schemas during application initialization or deployment preparation when possible. A bad generated schema should fail a release gate rather than produce a confusing field-level response on the first customer request. Request parsing remains a separate runtime boundary and needs its own payload-size and syntax policies.
Retain one valid control request beside each failure category. The control should parse, load the selected schema, validate, and reach the handler exactly once. Without that control, an adapter that rejects every request can satisfy an impressive collection of negative tests. The paired fixtures establish that the gate both stops the intended invalid input and permits a known legitimate operation.
Use category-specific assertions rather than one test expecting any exception. A catch-all can pass while the code raises a completely unrelated programming error. Confirm the intended exception type, and verify that invalid requests do not reach the handler. An unexpected exception should follow the application’s internal-error policy without exposing stack traces or being mislabeled as ordinary invalid input.
2. Do instance locations become correctly escaped JSON Pointers?
Convert path tokens individually into JSON Pointer reference tokens, preserving the empty root pointer and numeric array indexes. Test names containing a slash, a tilde, and an empty string. Do not join tokens into a dotted path or prepend a slash to every result, because those shortcuts change the location for valid but unusual JSON property names.
def pointer(tokens):
return "".join("/" + str(token).replace("~", "~0").replace("/", "~1")
for token in tokens)
assert pointer([]) == ""
assert pointer([""]) == "/"
assert pointer(["items", 2, "a/b~c"]) == "/items/2/a~1b~0c"
RFC 6901 defines the pointer syntax. The helper returns the plain pointer representation, not a URI fragment. If you later embed it in a fragment identifier, review that encoding layer separately rather than treating a hash prefix as the complete conversion.
Include an actual object with an empty-string member and another with a slash-bearing member in the test data. Checking only the pointer helper can miss an adapter that chooses the wrong path attribute before the helper runs. Resolve the resulting pointer against the original fixture and assert that it selects the expected existing value. Keep missing-property tests outside that resolution requirement because those values were never submitted.
For python-jsonschema, absolute_path identifies the location relative to the original instance. Keep schema locations separate. A path through properties and type is part of the schema, not the user’s request body. Swapping those paths can make a client try to highlight a field named properties that never existed in the submitted data.
Use the JSON Viewer to inspect sanitized input when a nested location surprises you. It helps reveal structure and array positions; it does not generate or validate your schema diagnostics. The JSON Pointer guide explains the difference between a concrete location and a query expression for selecting data.
3. Are missing-property errors attached to the containing object without inventing data?
For a required failure, preserve the containing object’s instance location and test missing fields separately from present-but-invalid fields. A missing property has no submitted value to resolve. If the public contract names missing fields separately, derive that information through a deliberate keyword-specific policy instead of scraping the validator’s English message or reporting a fabricated existing location.
schema = {"type": "object", "properties": {
"profile": {"type": "object", "required": ["email"]}
}}
errors = list(Draft202012Validator(schema).iter_errors({"profile": {}}))
assert len(errors) == 1
assert errors[0].validator == "required"
assert pointer(errors[0].absolute_path) == "/profile"
The error-handling documentation describes the path and rule information exposed by the library. For this fixture, profile exists and email does not. Reporting the parent location lets clients distinguish a missing member from a bad value at an existing member.
Compare that result with a separate schema requiring email to be a string and an instance containing numeric email. The latter failure belongs at the email location. Keep both fixtures so a reporting refactor cannot flatten every object error into a vague root-level failure or invent the same pointer for fundamentally different cases.
A product can expose a missing-property name for form convenience, but document whether that name identifies an absent field or a resolvable pointer. Be careful when generating several missing-field records from one required list: compare the list with the correct containing object, preserve special-character names, and avoid treating fields present with null as absent. Presence and value validity remain separate assertions.
4. Does a combined-schema failure retain useful branch context?
Test a failed anyOf or oneOf with its nested context rather than relying only on the top-level combination message. Keep branch identity and absolute instance locations available for internal diagnosis. A branch failure is evidence about that candidate schema, not automatically a repair instruction, because different alternatives can produce incompatible suggestions for the same input value.
schema = {"type": "object", "properties": {
"choice": {"anyOf": [{"type": "integer"}, {"const": "auto"}]}
}}
error = next(Draft202012Validator(schema).iter_errors({"choice": False}))
assert error.validator == "anyOf"
assert {child.validator for child in error.context} == {"type", "const"}
assert all(pointer(child.absolute_path) == "/choice" for child in error.context)
This failure means the value meets neither alternative. It does not mean the client must simultaneously send an integer and the string auto. A public message can describe the permitted alternatives while internal diagnostics preserve each candidate’s failed rule. Test that policy instead of exposing a raw recursive dump of every branch.
Nested context paths can be relative in library APIs, which is why the fixture uses absolute_path when identifying the original request location. Avoid manually appending a parent path to an already absolute child path. Such a bug often goes unnoticed until a combined schema appears beneath a nested object or array.
For oneOf, also distinguish no matching branches from multiple matching branches. The schema validation checklist introduces that ambiguity. An adapter that reports every oneOf failure as a missing field can encourage clients to add data that makes the overlap worse rather than resolving it.
5. Are public errors stable, deduplicated, and bounded by an explicit policy?
Normalize public diagnostics into approved location-and-code records, then apply a documented ordering, duplicate policy, and output limit. Test repeated failures at one location and failures at several locations. A stable public contract should not depend on incidental validator traversal order, and a response-size limit should not be described as a limit on the cost of validation itself.
def public_issues(errors, limit=10):
approved = {"type", "required", "enum", "const", "anyOf", "oneOf"}
records = {(pointer(e.absolute_path),
e.validator if e.validator in approved else "invalid")
for e in errors}
return [{"pointer": path, "code": code}
for path, code in sorted(records)[:limit]]
This illustrative adapter deduplicates by pointer and code and orders those pairs lexically. That is a chosen policy, not a schema requirement. It can intentionally collapse multiple failures with the same public identity; detailed internal diagnostics should remain available when the distinction matters to maintainers. A different product might preserve branch identifiers in its public contract.
Keep presentation text outside the stable machine identifier. A translated message can change without requiring a client release if clients branch on the documented code rather than comparing sentences. Test that localization does not alter pointers, public codes, or the acceptance decision. If you expose a request identifier for support, generate it through the normal application mechanism and verify that it does not embed request content or become part of the deduplication key.
Validate the limit as trusted application configuration. If the UI needs to know that more errors exist, add a deliberate truncation indicator rather than claiming the returned list is exhaustive. Test a zero-output configuration and a payload exceeding the normal limit so clients can handle that state predictably.
The helper collects all supplied errors before slicing them. Consequently, it does not bound validator work or memory for error collection. For large inputs, design resource limits and any early-stop strategy separately, then document whether globally sorted complete results are still promised. Do not hide a performance or completeness tradeoff behind an apparently harmless ten-item response cap.
6. Can the public response and diagnostic logs avoid exposing submitted secrets?
Test the complete error response and logging path with artificial sensitive values. Raw validator messages or exception strings can include instance data, so expose only reviewed fields and approved text. Also consider property names and pointers themselves: dynamic keys can contain sensitive identifiers, making a location-only response insufficient for some APIs without an additional redaction policy.
schema = {"type": "object", "properties": {"secret": {"const": "allowed"}}}
errors = list(Draft202012Validator(schema).iter_errors({"secret": "SENSITIVE_FIXTURE"}))
public = json.dumps({"errors": public_issues(errors)})
assert "SENSITIVE_FIXTURE" not in public
assert "allowed" not in public
assert json.loads(public)["errors"] == [{"pointer": "/secret", "code": "const"}]
This fixture proves only the shown adapter output excludes those values. It does not certify every response hook or log sink. Test the serialized HTTP body, application logs, and tracing integration using the same fake marker. A framework can accidentally serialize an exception after your helper has already built a safe error object.
OWASP’s logging guidance identifies data that should be excluded or handled carefully in logs. Apply the project’s access and retention rules to diagnostic detail; “internal” is not permission to retain complete submitted credentials indefinitely. The JSON redaction guide covers the related payload-handling boundary.
Finally, keep acceptance tests and reporting tests paired. A useful error must accompany a rejected request, and the rejected request must not execute business logic. After a dependency upgrade, verify location, public code, branch handling, limits, and data exposure again. That makes the reporting contract reproducible even when human-readable library messages evolve.