Quick answer: Check header names before mapping rows, distinguish missing cells from empty ones, and let the CSV parser handle quoted newlines. Decode the file using the agreed encoding and convert only fields with known types. A JSON serializer can write the resulting records, but it cannot recover data the import discarded.
A CSV can produce valid JSON while dropping one of two columns with the same name. These fixtures use Python 3.12.14 and a synthetic catalog export. Save the Python blocks in order as one file. The examples test import decisions rather than assuming that every CSV feed follows the same rules.
1. How can you catch duplicate headers before they overwrite values?
Read the header list and reject duplicate names before consuming the first record. A dictionary cannot hold two different values under one key, so waiting until you see the resulting JSON is too late. If you rename or trim headers, repeat the uniqueness check on the names the importer will use.
import csv
import io
import json
from contextlib import contextmanager
def stream(text):
return io.StringIO(text, newline="")
@contextmanager
def rejects(error):
try:
yield
except error:
pass
else:
raise AssertionError(f"expected {error.__name__}")
def check_headers(names):
if not names or any(not name for name in names):
raise ValueError("missing or empty header")
if len(set(names)) != len(names):
raise ValueError("duplicate header")
duplicated = csv.DictReader(stream("sku,sku\nA,B\n"))
assert duplicated.fieldnames == ["sku", "sku"]
assert next(duplicated) == {"sku": "B"}
with rejects(ValueError):
check_headers(duplicated.fieldnames)
with rejects(ValueError):
check_headers([name.strip() for name in ["sku", " sku"]])
check_headers(["sku", "label"])
The DictReader documentation explains how the header becomes dictionary keys. The first assertion shows that both names are visible before mapping, and the second shows that only one value remains afterward. In an import pipeline, check the names at the first point where both are still available. Report the conflicting names so the producer can fix the export rather than asking a reviewer to infer which column was lost.
Decide whether header order matters to your contract. If the code maps by name and column reordering is allowed, check the set of accepted names and reject repeats separately. If positional conventions matter, compare the ordered sequence as well. A set comparison alone would miss a repeated header when all required names happen to be present. The CSV to JSON guide introduces the header mapping step; this fixture tests whether the mapping can represent the source faithfully.
A compatibility mapping may rename an old column to a new name. Test a file containing both names before allowing that mapping in production. Otherwise a transitional export can collapse two distinct source columns into one destination field even though each original header was unique. An explicit rejection makes the producer resolve which value is authoritative.
For the header test, use different values in the repeated columns. If both cells contain the same text, a test that compares just the surviving value cannot show which input was lost. Also cover the empty file and a header without records if your workflow distinguishes them. An empty file has no header to validate; a header without records may be a legitimate empty export. Make that distinction before deciding whether the job should publish an empty JSON array.
2. How do you distinguish a short row from a present but empty cell?
Give missing fields a sentinel that cannot occur as a parsed string, and use a separate marker for excess fields. Reject or report either condition before writing JSON. A trailing delimiter can create a present empty cell, whereas a short row can omit the field completely. Those cases often need different policies.
missing, extra = object(), object()
reader = csv.DictReader(
stream("sku,label\nA,\nB\nC,Box,unexpected\n"),
restval=missing, restkey=extra,
)
check_headers(reader.fieldnames)
empty, short, long = list(reader)
assert empty["label"] == ""
assert short["label"] is missing
assert long[extra] == ["unexpected"]
def check_width(row):
if extra in row or any(value is missing for value in row.values()):
raise ValueError("unexpected row width")
check_width(empty)
for row in (short, long):
with rejects(ValueError):
check_width(row)
The Python CSV reference documents the two padding options. Using a Python object as the marker keeps it distinct from any real column name or string value in the fixture. These markers belong to the import logic. They must never be included in a JSON payload: the extra marker is not a valid JSON key, and the missing marker is not a serializable field value.
An importer that accepts an empty label might still reject a missing label column. In the first case the producer supplied the expected shape and left a value blank. In the second it supplied a different row width. The distinction can matter when diagnosing an export that lost a delimiter. Avoid filling both cases with a default before recording the difference. The JSON Schema guide covers downstream constraints, but those checks cannot tell how many CSV cells were present once an importer has normalized them into identical objects.
Keep a valid row in the fixture with the same delimiter and header as the malformed rows. That helps separate a row width problem from an importer configured for the wrong delimiter. In a real batch, decide whether rejecting a single record stops the whole import or queues the record for review. Test that policy at the point where writes actually happen.
3. How do you test quoted newlines without confusing lines and records?
Use a fixture with a quoted field containing both a comma and a newline, followed by another record. Read it through the CSV parser and check both the field contents and the number of records. Do not split the file into lines and parse each line independently when fields may span lines.
reader = csv.DictReader(stream('sku,label\r\nA,"Box, large\r\nBlue"\r\nB,Plain\r\n'))
check_headers(reader.fieldnames)
assert next(reader) == {"sku": "A", "label": "Box, large\r\nBlue"}
assert reader.line_num == 3
assert list(reader) == [{"sku": "B", "label": "Plain"}]
assert reader.line_num == 4
The CSV module documentation recommends opening file objects with newline handling left to the parser. The fixture preserves the embedded carriage return and line feed and checks them directly. The line counter includes the header and the physical lines used by a record. It is useful for locating a problem in a source file, while a separate record index is useful for identifying the affected object in an export.
For a visual comparison, try the synthetic multiline fixture in the CSV to JSON Converter. It handles quoted fields and can help you inspect the resulting object shape. It uses its own delimiter detection and value conversion rules, so use the Python assertions to verify your import contract. For background on the format and its dialects, see Wikipedia's CSV overview.
When a support report says that row three failed, record what that number means. A spreadsheet, text editor, and import log can count differently once a field spans lines. Consider reporting both a record index and the last physical line consumed, alongside a source identifier that lets the operator find the right file.
Include a genuine blank physical line outside any quoted field if your source permits one. A blank line within a quoted cell belongs to the cell, while an empty physical row outside it can be skipped by DictReader. A comma on its own is a different input again: it represents two empty cells rather than no cells. Test whichever interpretation the importer promises, and avoid using a raw count of newline characters as the number of imported records.
4. How do you keep a UTF-8 BOM out of the first JSON key?
Decode the file as UTF-8 with BOM handling when the source contract permits an initial UTF-8 signature. Test the same header with and without that signature, then check the exact resulting names. Ordinary UTF-8 decoding can leave an invisible character attached to the first key even though the text looks correct.
plain = b"sku,label\nA,Box\n"
signed = b"\xef\xbb\xbf" + plain
assert csv.DictReader(stream(signed.decode("utf-8"))).fieldnames == [
"\ufeffsku", "label",
]
for data in (plain, signed):
reader = csv.DictReader(stream(data.decode("utf-8-sig")))
assert reader.fieldnames == ["sku", "label"]
assert next(reader)["sku"] == "A"
The Python codecs documentation describes UTF-8 signature handling. This fixture tests bytes because a text editor may consume the signature before you copy the text. It compares exact names so a visually similar key cannot pass. In a file based importer, apply the same encoding policy when opening the file, and use explicit newline handling as in the preceding fixture.
Treat encoding as part of the feed contract. Accepting an optional UTF-8 BOM does not mean the reader should guess every other encoding. If a partner sends a different encoding, document it and add a fixture with a character that would decode incorrectly under UTF-8. Avoid silently replacing malformed bytes in identifiers, since the replacement can make distinct source values look the same.
If only the first field appears to be missing while later fields are present, inspect the decoded header representation before rewriting the column mapping. A mismatch can come from the input decoder rather than a renamed field. Fixing the encoding at its boundary avoids adding a special alias for a name that was never meant to exist.
When debugging an encoding change, keep a fixture with a non-ASCII label as well as the ASCII headers shown here. A decoder can produce the expected keys while corrupting the label text, especially if a fallback decoder accepts bytes that the intended decoder would reject. Compare the label after the JSON round trip and let decoding errors surface in the import report rather than silently rewriting the source.
5. How do you verify that JSON conversion preserves identifiers and types?
Define conversion rules for each field before serializing the record. Keep identifiers as text when leading zeros carry meaning, convert known counts explicitly, and map empty cells to null only where the contract calls for it. Check the decoded JSON values rather than trusting that a successful write preserved the intended types.
reader = csv.DictReader(stream("sku,quantity,note\n0007,2,\n"))
assert reader.fieldnames == ["sku", "quantity", "note"]
row = next(reader)
result = json.loads(json.dumps({
"sku": row["sku"],
"quantity": int(row["quantity"]),
"note": None if row["note"] == "" else row["note"],
}))
assert result == {"sku": "0007", "quantity": 2, "note": None}
assert type(result["quantity"]) is int
The conversion is intentionally narrow. It assumes the width has already been checked and does not enforce a business range for the quantity. Python's integer conversion also accepts inputs such as surrounding whitespace. If the feed requires a stricter lexical representation or a nonnegative range, define and test those requirements before converting. Keep the lexical check separate from verifying the output type so a later normalization cannot mask invalid source text.
A stable test suite covers more than one happy path. Use a zero quantity to catch a fallback that mistakes zero for missing, and an identifier made entirely of digits to catch blanket number inference. If a field should be omitted instead of sent as null, assert key presence explicitly. Read the JSON versus CSV guide for how those differences arise when moving from a table to typed JSON.
Finally, compare source and destination record counts when the producer gives you an independent expected count. A schema can tell you that each exported record has an appropriate shape, but it cannot prove the import included every row. Counts, field assertions, and deliberately malformed source fixtures answer different questions. Keep each check close to the stage where it still has the evidence needed to explain a failure.