JSON Encoding Errors: UTF-8, BOM, and the Invisible Character That Breaks Parsing

Quick answer: JSON must be UTF-8, and it must not start with a byte order mark (BOM). If a parser reports an unexpected token at position 0 on a file that looks fine, the file almost certainly begins with the invisible BOM character U+FEFF (bytes EF BB BF). Save the file as "UTF-8" rather than "UTF-8 with BOM", or strip it in code: Python open(path, encoding="utf-8-sig"), Node text.replace(/^\uFEFF/, ""). Garbled text like é is a different bug: UTF-8 bytes being read as Latin-1.

The file opens in Notepad. It is clearly valid: an opening brace, well-formed keys, no trailing comma. You paste it into a validator and it passes. Then JSON.parse throws at position 0, the very first character, which is a brace. Except it is not. There is a character in front of the brace that no editor shows you, and it was put there by the tool that saved the file.

After addressing the encoding or serialization issue, inspect a valid JSON sample in the JSON Viewer.

What the Spec Requires: UTF-8, No BOM

RFC 8259 section 8.1 is unusually blunt for a standards document. JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8. Implementations MUST NOT add a byte order mark to the beginning of a networked-transmitted JSON text. And, in the interest of interoperability, parsers MAY ignore a BOM rather than treating it as an error.

That last clause is why the problem is so inconsistent. Some parsers are lenient and skip the BOM; others are strict and choke. The same file can work in one program and fail in the next, which sends people looking for a bug in the wrong place.

What a BOM Is and Why It Is Invisible

A byte order mark is the Unicode character U+FEFF placed at the start of a file. In UTF-16 it tells the reader which byte order the file uses. In UTF-8 there is no byte order to signal, so the BOM serves only as a "this is UTF-8" hint, and most Unix tools treat it as garbage. Encoded in UTF-8 it is three bytes: EF BB BF.

Editors hide it because it is defined as a zero-width no-break space. Terminals usually hide it too. So you get a file that looks like it starts with { but actually starts with U+FEFF {. Every error below comes from that one invisible character or from its cousins in the encoding family.

Symptoms: The Error Messages, by Language

The exact wording varies, but the pattern is an error at the very start of the input. Here is what a BOM-prefixed file produces:

// Node.js / Chrome (JSON.parse on a string that starts with U+FEFF)
SyntaxError: Unexpected token '\ufeff', "\ufeff{"name": "..." is not valid JSON
// older V8 wording:
SyntaxError: Unexpected token  in JSON at position 0

# Python
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

# Go (encoding/json)
invalid character 'ï' looking for beginning of value

Python is the friendliest: it names the problem and the fix in the message. Go shows the first byte of the BOM interpreted as Latin-1, which is where the ï comes from. The older Node message is the nastiest because the token between "token" and "in" is the BOM itself, rendered as nothing. If you ever see an error message with what looks like two spaces in a row, that is your clue. The unexpected token guide maps the other first-character cases, such as < for an HTML response.

A second family of symptoms is mojibake: the file parses, but string values come out wrong.

The tell is that every non-ASCII character becomes two or three characters starting with Ã or â. That is UTF-8 being decoded as Windows-1252 or ISO-8859-1. The reverse mistake, Latin-1 bytes decoded as UTF-8, produces � replacement characters or a hard decode error instead.

Where the BOM Comes From

You almost never add a BOM on purpose. These are the usual culprits:

• Windows Notepad (older versions). For years the "UTF-8" choice in Notepad wrote a BOM. Recent Windows 10 and 11 builds default to UTF-8 without BOM, but files saved years ago and machines with the old default still produce them.
• Windows PowerShell 5.1. Out-File -Encoding utf8 and Set-Content -Encoding utf8 write a BOM. Plain > redirection in 5.1 is worse: it writes UTF-16LE. PowerShell 7 fixed both; its utf8 means no BOM and there is a separate utf8BOM option.
• Excel's "CSV UTF-8" export. Adds a BOM so that Excel itself can reopen the file correctly. If you then convert that CSV to JSON with a naive script, the BOM ends up glued to the first header name. The CSV to JSON guide covers that specific trap.
• .NET. File.WriteAllText(path, text, Encoding.UTF8) emits a BOM because that encoding object has a preamble. new UTF8Encoding(false) does not.
• Legacy Visual Studio and some Windows IDEs that defaulted to UTF-8 with signature.

How to Detect a BOM

Stop guessing and look at the bytes. On macOS or Linux:

$ xxd config.json | head -1
00000000: efbb bf7b 0a20 2022 6e61 6d65 223a 2022  ...{.  "name": "

$ file config.json
config.json: UTF-8 Unicode (with BOM) text

$ head -c 3 config.json | od -An -tx1
 ef bb bf

On Windows, PowerShell can read the first three bytes directly:

[IO.File]::ReadAllBytes("config.json")[0..2] | ForEach-Object { $_.ToString("X2") }
# EF
# BB
# BF

In VS Code, the encoding is shown at the right end of the status bar. "UTF-8 with BOM" is the smoking gun; "UTF-16 LE" is the PowerShell redirection case.

One more quick check: paste the content into JSON Prettifier. A validation error at line 1, column 1 on a document whose first visible character is { or [ is the BOM, since nothing else can fail before the first brace.

How to Fix It

Save the file correctly

In VS Code: click the encoding in the status bar, choose Save with Encoding, pick UTF-8. To stop it happening again, set "files.encoding": "utf8" in settings; VS Code only writes a BOM when you explicitly pick "utf8bom". In Notepad, use "UTF-8" not "UTF-8 with BOM" in the Save As encoding dropdown.

Strip it in Node.js

fs.readFileSync(path, "utf8") keeps the BOM as a leading U+FEFF character. Remove it before parsing:

const fs = require("fs");

const raw = fs.readFileSync("config.json", "utf8");
const text = raw.charCodeAt(0) === 0xfeff ? raw.slice(1) : raw;
const config = JSON.parse(text);

Or the regex form, raw.replace(/^\uFEFF/, ""). Note that require("./config.json") works even with a BOM because Node's module loader strips it before parsing, which is one reason the same file behaves differently depending on how you load it. The reading and writing JSON files in Node post compares the loading methods.

Strip it in Python

Python ships an encoding that does it for you:

import json

with open("config.json", encoding="utf-8-sig") as f:
    config = json.load(f)

# Already have the string? Strip the character:
text = text.lstrip("\ufeff")

utf-8-sig reads plain UTF-8 and UTF-8 with BOM identically, so it is safe to use everywhere you read files you did not write.

Strip it on the command line

# In place, GNU sed
sed -i '1s/^\xEF\xBB\xBF//' config.json

# Or skip the first three bytes into a new file
tail -c +4 config.json > clean.json

# Convert a UTF-16 file (PowerShell 5.1 redirection) to UTF-8
iconv -f UTF-16 -t UTF-8 config.json > clean.json

Writing without a BOM from PowerShell 5.1, which has no built-in option for it:

$utf8NoBom = New-Object System.Text.UTF8Encoding($false)
[IO.File]::WriteAllText("C:\data\config.json", $json, $utf8NoBom)

Fix mojibake

If the garbling is in the pipeline, fix the reader: open with encoding="utf-8" in Python (never rely on the platform default, which on Windows may still be a legacy code page), and pass "utf8" to readFileSync in Node. If the damage is already stored, reverse the wrong decode:

# Python: "Café" -> "Café"
fixed = broken.encode("cp1252").decode("utf-8")

// Node
const fixed = Buffer.from(broken, "latin1").toString("utf8");

This only works if the original bytes survived the round trip. If a ? or � replaced the character, the information is gone and you need to re-export from the source.

Over HTTP: set the Content-Type

Serve JSON as application/json. The charset parameter is technically meaningless for JSON because UTF-8 is mandated, but a legacy intermediary that sees no charset may assume Latin-1 and transcode. Most frameworks (Express res.json, Flask jsonify) already set application/json; charset=utf-8. Per the Fetch specification, browsers decode a response body with the UTF-8 decode algorithm, which drops a leading BOM, so response.json() tolerates a BOM that JSON.parse(fs.readFileSync(...)) does not. The content-type details are in the application/json guide.

Gotchas

• A BOM in the middle of a file. Concatenating two BOM-prefixed files puts a U+FEFF in the middle, which is a hard syntax error. Strip before concatenating.
• BOM inside a string value. If a CSV header carried a BOM into a JSON key, the key becomes "\ufeffname", not "name". Lookups silently return undefined. Check Object.keys(obj)[0].length.
• Escaped vs. literal. \uFEFF written as six ASCII characters inside a JSON string is valid and harmless; only the raw byte sequence at the start of the file is the problem. The escape characters guide covers how \u sequences work.
• Git and diffs. A BOM shows up in diffs as a changed first line with no visible change. Add *.json text eol=lf to .gitattributes and reject BOMs in CI with the head -c 3 check above.
• ensure_ascii as a workaround. Python's json.dumps escapes non-ASCII by default, which sidesteps mojibake at the cost of unreadable output. It does nothing about a BOM.