Quick answer: Run six JSON Unicode tests before comparing API strings: distinguish escape spelling from decoded values, test canonical normalization, review compatibility folding, detect normalized-key collisions, define length units, and reject unsupported surrogate sequences. Visual similarity is not an equality policy. Preserve original values wherever identifiers, signatures, or external contracts require exact data.
Two API responses display the same customer label, yet a strict comparison reports a change. Another comparison becomes green after normalization—but two formerly distinct keys now collide. Both failures come from treating text appearance as a specification.
The JSON Escape tool helps inspect escape layers using JSON string rules. It is not a normalization engine or Unicode validator. Our JSON escape-character guide covers syntax; these tests concern the decoded values that reach application code. Wikipedia's Unicode equivalence overview supplies background terminology.
The six independent JavaScript examples below were executed on Node.js 24.19.0. The first uses this repository's escape helper and requires its checkout; the others use JavaScript APIs available in the tested runtime. These are focused regression fixtures, not a complete Unicode conformance suite, a spoofing detector, or a cross-runtime compatibility certification.
1. Do different escape spellings decode to the same value?
Compare decoded string values separately from their JSON source spelling. A literal character and its Unicode escape can represent the same value, while a doubled backslash can represent literal escape-looking text. Parse exactly the number of layers specified by the protocol instead of repeatedly unescaping until the display looks familiar or a comparison happens to pass.
Start by identifying what the API sends: a JSON document containing a string, or a string containing another JSON document. Those are different protocols. One parsing step should not silently become two because an engineer recognizes a backslash sequence in the output.
RFC 8259 describes JSON string escapes and string comparison. An escaped spelling does not make a character more or less equivalent than its literal spelling after decoding. That question is separate from whether two different Unicode sequences are canonically equivalent.
import assert from 'node:assert/strict';
import { escapeJsonString, unescapeJsonString } from './lib/json-tools/escape.mjs';
const escapedSource = String.raw`"caf\u00e9"`;
const literalSource = '"café"';
assert.notEqual(escapedSource, literalSource);
assert.equal(JSON.parse(escapedSource), JSON.parse(literalSource));
const literalEscapeText = String.raw`\u00e9`;
assert.equal(unescapeJsonString(escapeJsonString(literalEscapeText)), literalEscapeText);
assert.notEqual(literalEscapeText, 'é');
Keep both the source text and the parsed value when diagnosing an encoding report. A log formatter can add another escape layer, making the same value appear different in logs and a browser. Do not change production data based solely on the representation a debugging interface chooses to show.
If the contract requires comparing original bytes, decoding first answers a different question. Our JSON canonicalization checks explain why byte identity and value equality must not be mixed. A successful string round trip is useful evidence about the tested layer, not permission to rewrite authenticated input.
2. Do canonically equivalent sequences follow the field's normalization policy?
Test a precomposed character and its decomposed combining sequence after JSON parsing. They can look alike while remaining unequal under exact string comparison. If the field permits canonical normalization, apply the same documented form at the agreed boundary and test both matching equivalents and unrelated values that must continue to differ.
Unicode Normalization Form C, or NFC, provides canonical composition. Normalization Form D, or NFD, provides canonical decomposition. The Unicode normalization specification defines these forms. Choosing one for an application remains a contract decision; JSON syntax itself does not choose for you.
import assert from 'node:assert/strict';
const composed = JSON.parse(String.raw`"\u00e9"`);
const decomposed = JSON.parse(String.raw`"e\u0301"`);
assert.notEqual(composed, decomposed);
assert.equal(composed.normalize('NFC'), decomposed.normalize('NFC'));
assert.equal(composed.normalize('NFD'), decomposed.normalize('NFD'));
assert.notEqual(composed.normalize('NFC'), 'e'.normalize('NFC'));
The final assertion matters. A normalizer that strips accents entirely could make a positive fixture pass while implementing a broader transformation than intended. Include negative controls so the test establishes the actual boundary rather than merely proving that one pair eventually matches.
For a display name, a normalized search key may be useful while the original spelling remains stored for display. For an opaque token or external identifier, any rewrite may be forbidden. Keep those fields out of a blanket recursive normalizer unless their owners have explicitly approved the policy.
Define the boundary consistently across producers and consumers. Normalizing only in a test comparator can hide a production mismatch in a database lookup or cache key. Conversely, introducing normalization into existing stored keys needs a migration and collision review. Treat it as a data-model change, not a cosmetic cleanup to make a diff smaller.
3. Would compatibility normalization erase a distinction the API needs?
Test compatibility normalization separately from canonical normalization. Compatibility forms can collapse additional distinctions that an application may need to preserve, so they should not be a default fix for every unequal string. Document the field's permitted equivalences and verify that identifiers, display text, and search representations are handled under their own rules.
Unicode Normalization Form KC, or NFKC, includes compatibility composition. It is not simply a stronger spelling of NFC with universally better results. MDN's normalize documentation explains the supported forms and examples of their different effects.
import assert from 'node:assert/strict';
const circled = '\u2460';
assert.notEqual(circled.normalize('NFC'), '1');
assert.equal(circled.normalize('NFKC'), '1');
assert.equal('Ａ'.normalize('NFKC'), 'A');
assert.notEqual('A'.normalize('NFC'), 'a'.normalize('NFC'));
This example shows why a display label and a normalized lookup key can have different requirements. A product code containing a circled digit might be intentionally distinct, while a search feature may deliberately match it to an ordinary digit. Neither policy should be inferred from the fact that both inputs are strings.
Normalization also does not automatically establish case-insensitive equality or remove every visually confusing character. Keep case handling, language-sensitive search, and spoofing defenses as separately specified behaviors. Do not market a short normalize call as a complete security solution for usernames or internationalized identifiers.
Keep the original value when deriving a compatibility-based search representation, subject to the application's retention rules. This makes it possible to explain what matched and to change the search policy later. If only the rewritten value survives, distinctions erased by the chosen form cannot be reconstructed from that rewritten value alone.
Review fixtures with domain owners before adding compatibility rules. The useful question is not whether two glyphs look close enough to an engineer; it is whether the application intentionally treats their values as interchangeable in this particular field and operation.
4. Can normalizing property names cause a collision?
Construct an object containing two distinct keys that normalize to the same name, then require collision detection before reconstruction. Normalizing keys is a transformation of field identity, not only string presentation. Reject ambiguity or define an explicit merge policy rather than allowing assignment order to decide silently which value survives in a rebuilt object.
Value normalization and key normalization have different consequences. Two customer labels becoming equal may affect search; two object keys becoming equal can remove the ability to address both fields independently. A recursive convenience function can therefore change structure even when every individual normalize call behaves correctly.
import assert from 'node:assert/strict';
const source = JSON.parse(String.raw`{"\u00e9":1,"e\u0301":2}`);
assert.equal(Object.keys(source).length, 2);
const normalized = Object.keys(source).map(key => key.normalize('NFC'));
assert.equal(new Set(normalized).size, 1);
const requireUniqueNormalizedKeys = object => {
const names = Object.keys(object).map(key => key.normalize('NFC'));
assert.equal(new Set(names).size, names.length, 'normalized-key collision');
};
assert.throws(() => requireUniqueNormalizedKeys(source));
requireUniqueNormalizedKeys({ first: 1, second: 2 });
This check only examines the demonstrated object's own keys. It is not a recursive schema validator or a safe general-purpose object rebuilder. If nested mappings are allowed, define which paths are normalized and test collisions at those paths before introducing a recursive transformation.
Our duplicate JSON key guide covers the related parsing hazard. Here the original keys are different; the collision is introduced by the application after parsing. That distinction explains why a parser accepting the original document cannot certify that later normalization is lossless.
For an external schema, prefer preserving its field names exactly unless the schema defines another rule. For application-controlled keys, consider validating an approved naming policy at creation rather than rewriting arbitrary keys during comparison. If the policy changes, inventory stored names and identify collisions before migration so a successful deployment cannot conceal overwritten data.
5. Does each length limit name the unit it actually measures?
Test UTF-16 code units, Unicode code points, grapheme clusters, and UTF-8 bytes as separate measurements. A limit called characters is ambiguous unless the API defines the unit. Use representative combining sequences and emoji, then apply the same measurement at validation and truncation boundaries so accepted values are not unexpectedly cut apart later.
MDN documents JavaScript string length as a count of UTF-16 code units. A code-point iterator answers a different question. Grapheme segmentation addresses user-perceived text clusters, while a byte budget concerns an encoding of the string rather than its appearance.
import assert from 'node:assert/strict';
const text = 'e\u0301';
const segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' });
assert.equal(text.length, 2);
assert.equal([...text].length, 2);
assert.equal([...segmenter.segment(text)].length, 1);
assert.equal(new TextEncoder().encode(text).length, 3);
assert.equal('😀'.length, 2);
assert.equal([...'😀'].length, 1);
The Intl.Segmenter documentation describes segmentation options. These small assertions verify the tested strings in the tested runtime; they do not guarantee identical segmentation for every Unicode sequence across different runtime data versions. Record the runtime when cross-service agreement matters.
Do not truncate a JavaScript string at an arbitrary code-unit index and assume the result is an intact user-visible character. Decide whether truncation operates on graphemes, code points, or bytes, and use an implementation suited to that contract. Test the output again for well-formedness and for the receiver's actual limit.
Keep encoding failures separate from normalization differences. The UTF-8 and BOM guide covers the transport boundary. A normalization operation cannot repair text that was already decoded under the wrong encoding, and counting bytes after such damage does not establish that the original value survived.
6. Are unsupported lone surrogates rejected before downstream use?
Include an escaped lone surrogate and a valid surrogate pair in parsing tests, then check the decoded string's well-formedness explicitly. A JSON parser accepting escape syntax does not establish interoperability with every downstream encoder or API. Reject unsupported values or apply a documented replacement policy rather than silently treating replacement as recovery of the original text.
The JSON specification notes interoperability problems with unpaired surrogates. JavaScript strings can contain such code units, which makes a separate check useful at boundaries requiring well-formed Unicode. MDN's isWellFormed documentation describes the relevant JavaScript predicate.
import assert from 'node:assert/strict';
const lone = JSON.parse(String.raw`"\ud800"`);
const paired = JSON.parse(String.raw`"\ud83d\ude00"`);
assert.equal(lone.isWellFormed(), false);
assert.equal(paired.isWellFormed(), true);
assert.equal(lone.normalize('NFC').isWellFormed(), false);
assert.equal(lone.toWellFormed(), '\ufffd');
assert.notEqual(lone.toWellFormed(), lone);
Replacing a malformed sequence with the replacement character changes data. That may be acceptable for a display-only fallback, but it is not proof of what the sender intended and should not happen unnoticed in identifiers or authenticated values. Decide which fields reject input and which can expose a labeled display fallback.
Keep a negative test where malformed text reaches the specific validation boundary. A fixture that fails earlier because of a missing required field does not prove the Unicode check ran. Also test already valid non-ASCII text so a defensive rule does not accidentally reject every character outside a narrow alphabet.
Finish by running the same fixtures through actual producers and consumers before changing a shared string policy. Store normalization forms, length units, runtime versions, rejection rules, and any permitted replacement behavior with the API contract. Escaping, comparing, validating, and signing are different operations; reliable tests make those boundaries visible instead of using one transformation to stand in for all four.