Skip to content
JavaScript

6 Node.js Fetch Gzip JSON Tests for HTTP Proxies

·10 min read
6 Node.js Fetch Gzip JSON Tests for HTTP Proxies - JavaScript guide from JSON Prettifier

Quick answer: A Node.js proxy must send headers that describe the bytes it actually forwards. If Fetch has decoded gzip, remove the upstream encoding label and handle the decoded length separately. If the proxy must preserve gzip, use a client interface that exposes the original bytes and verify them at the downstream boundary.

The examples run on Node.js v24.22.1 with a disposable loopback server and synthetic JSON. Save the JavaScript blocks in order as one file named proxy-fixture.mjs and run it with Node.

1. How do you tell whether Fetch returned decoded JSON bytes?

Serve a known gzip payload over HTTP and compare the bytes Fetch exposes with both the compressed fixture and the original JSON text. This establishes what the network client gives your proxy. Inspecting the upstream header alone cannot tell you whether the client has already transformed the body.

import assert from 'node:assert/strict';
import { once } from 'node:events';
import { createServer, get } from 'node:http';
import { gzipSync } from 'node:zlib';

const original = Buffer.from(JSON.stringify({ note: 'ready'.repeat(200) }));
const zipped = gzipSync(original);
const corrupt = Buffer.from('not a gzip member');
const upstream = createServer((req, res) => {
  const body = req.url === '/corrupt' ? corrupt
    : req.url === '/invalid-json' ? gzipSync('{') : zipped;
  res.writeHead(200, {
    'content-type': 'application/json',
    'content-encoding': 'gzip',
    'content-length': body.length,
  });
  res.end(body);
});
upstream.listen(0, '127.0.0.1');
await once(upstream, 'listening');
upstream.unref();
const url = 'http://127.0.0.1:' + upstream.address().port;
const first = await fetch(url);
const decoded = Buffer.from(await first.arrayBuffer());
assert.deepEqual(decoded, original);
assert.notDeepEqual(decoded, zipped);

The Node Fetch reference identifies the implementation as based on Undici. The Node zlib reference describes the gzip helper used to construct the wire fixture. For background on the compressed format, see Wikipedia's gzip overview. Compression changes the representation of the document. It does not change the JSON fields a consumer should get after decoding.

A constructor based mock can be useful for business logic, but it does not put bytes through the HTTP stack. If your project uses a custom dispatcher or a different runtime, repeat the real response test there. The fixture checks one gzip encoding and a normal success status. It does not establish the behavior of redirects, streaming uploads, or stacked content encodings. For broader endpoint tests, see the JSON API compression guide.

When a bug report includes a response saved by a client, ask how the capture was made. A packet capture, an HTTP client's raw stream, and the text shown in a debugger can represent different stages. Save the two representations under different names in the fixture and record which one each assertion uses. A byte comparison is useful here precisely because it does not depend on how a debugger renders the contents.

2. Which encoding header should a proxy send with a decoded body?

Remove Content-Encoding when the outgoing body is the decoded JSON and no other encoding was applied. Keep the JSON media type if the content is still JSON. Test the downstream envelope by pairing the actual outgoing bytes with the headers the proxy plans to send, rather than copying upstream metadata blindly.

assert.equal(first.headers.get('content-encoding'), 'gzip');
const headers = new Headers(first.headers);
headers.delete('content-encoding');
assert.equal(headers.get('content-encoding'), null);
assert.equal(headers.get('content-type'), 'application/json');
assert.deepEqual(decoded, original);

The HTTP Content-Encoding rules describe the coding applied to the representation. In this fixture the header available through Fetch still says gzip, although the body it exposes is plain JSON. A downstream client that receives those plain bytes with the old label can fail in its decoding layer, before it reaches a JSON parser. Inspect the bytes and the header together when diagnosing that symptom.

The header copy above is deliberately incomplete and used only to isolate one mismatch. Real proxy code should construct a deliberate downstream header set. Caching metadata and representation validators can also become inappropriate when the body changes, and connection specific headers have separate forwarding rules. Give the rewrite its own test rather than treating the origin header list as a safe template. If an outer framework compresses the response again, test its final output on the wire.

Apply the same rule if you rewrite the JSON itself. Removing the gzip label corrects one transformation; changing the JSON fields introduces another. Test the actual response serializer and compression middleware in the order they run. If another layer owns compression, it should set the encoding label when it applies that encoding. The result the next client sees is the one that matters, even when the handler returned the expected object.

3. How do you detect a stale Content-Length after decoding?

Compare the upstream Content-Length with the compressed fixture, then compare the outgoing length with the actual bytes you intend to send. Recompute the length if you have a complete buffered body. If you stream it, let the server framework frame the response instead of reusing the upstream compressed length.

assert.equal(Number(first.headers.get('content-length')), zipped.length);
assert.notEqual(zipped.length, decoded.length);
headers.set('content-length', String(decoded.length));
assert.equal(Number(headers.get('content-length')), original.length);

HTTP Content-Length is expressed in octets. JavaScript string length is the wrong measurement when the outgoing byte encoding differs from the string representation. If you serialize a new object, measure the final encoded buffer or let the framework handle framing. The test uses deliberately repetitive text so the mismatch between compressed and decoded lengths is easy to observe.

A proxy may appear to work with a small sample when lengths happen to be similar or a client ignores a bad value. Keep a fixture where the sizes clearly differ, then test a real downstream read to confirm the consumer receives the whole document. A correct JSON parser cannot repair HTTP message framing. Inspect the declared length and actual outgoing bytes before investigating why a body appears truncated.

To exercise the decoded response over a second HTTP connection, send the corrected headers and body from a small downstream server. This closes the gap between inspecting a Headers object and observing what a consumer receives. The following fixture compares the complete payload after the downstream client consumes it. It uses only the controlled fixture headers from above, not a general purpose forwarding policy.

const downstream = createServer((_req, res) => {
  res.writeHead(200, Object.fromEntries(headers));
  res.end(decoded);
});
downstream.listen(0, '127.0.0.1');
await once(downstream, 'listening');
downstream.unref();
try {
  const reply = await fetch('http://127.0.0.1:' + downstream.address().port);
  assert.equal(reply.headers.get('content-encoding'), null);
  assert.equal(Number(reply.headers.get('content-length')), decoded.length);
  assert.deepEqual(Buffer.from(await reply.arrayBuffer()), original);
} finally {
  await new Promise(resolve => downstream.close(resolve));
}

4. How do you preserve gzip when the proxy must forward encoded bytes?

Choose an HTTP client that exposes the original compressed bytes and compare them with the upstream fixture. Preserve the gzip label and matching length when those bytes are forwarded unchanged. Verify the downstream body separately; successfully parsing JSON at the proxy does not prove that it preserved the encoded representation.

const wire = await new Promise((resolve, reject) => {
  get(url, res => {
    const chunks = [];
    res.on('data', chunk => chunks.push(chunk));
    res.on('error', reject);
    res.on('end', () => resolve({
      body: Buffer.concat(chunks), headers: res.headers,
    }));
  }).on('error', reject);
});
assert.deepEqual(wire.body, zipped);
assert.equal(wire.headers['content-encoding'], 'gzip');
assert.equal(Number(wire.headers['content-length']), wire.body.length);

This fixture uses the Node HTTP client, whose response body in this test contains the encoded data. It buffers the fixture for comparison. A production pass through can stream bytes, but should still check what happens when the upstream closes early and when the downstream client disconnects. Test the actual client configuration; changing libraries can change decoding behavior.

Keeping gzip bytes and decoding gzip bytes are separate modes. A service that needs to examine or modify the JSON before forwarding it may prefer the decoded path, then let its server framework decide whether to compress again. A service that only passes through a response can avoid that transformation. Decide which behavior the route promises before choosing the client. A switch between the two should update tests for both the payload and the metadata that accompanies it.

A useful extra integration case is an upstream error response whose body is also compressed. Test it separately if your proxy forwards upstream status codes. The body contract and the status contract can fail independently: a route might preserve the encoded bytes perfectly while incorrectly returning success to the caller. Record the expected status explicitly so a future refactor cannot hide a failure behind an otherwise correct payload.

5. How do you distinguish broken gzip from valid gzip containing invalid JSON?

Use two fixtures: bytes labeled gzip that cannot be decoded, and valid gzip containing text that cannot be parsed as JSON. In the first case, reading the response body fails. In the second, reading succeeds and JSON parsing fails. A successful HTTP status cannot distinguish either failure from a usable payload.

const broken = await fetch(url + '/corrupt');
assert.equal(broken.status, 200);
await assert.rejects(broken.arrayBuffer());
const malformed = await fetch(url + '/invalid-json');
const malformedText = await malformed.text();
assert.equal(malformedText, '{');
assert.throws(() => JSON.parse(malformedText), SyntaxError);

Keep those stages distinct in error reports. An upstream can send success headers before the client discovers a body problem. A decompression failure points toward the bytes or encoding label; a parse failure after successful decoding points toward document syntax or an unexpected text response. See the Fetch JSON body tests for related cases involving stream consumption and preserving the original error.

Paste a synthetic decoded document into the JSON Viewer tree when you need to inspect the object shape after parsing. The viewer works on the document you supply. It cannot inspect HTTP headers or tell you whether the upstream bytes were valid gzip. For operational diagnostics, record the failed stage without automatically logging the full response, which could contain credentials or customer data.

If your framework starts sending a success response before it knows whether the upstream body is readable, you may no longer be able to replace that status after a decoding failure. Test that scenario at the framework boundary. For a buffered route, verify that the proxy sends the failure response instead of the incomplete candidate. For a streaming route, verify how the downstream client detects early termination.

6. Where should a proxy enforce a decoded JSON size limit?

Count bytes as the decoded stream is consumed and stop reading when the application limit is exceeded. A compressed Content-Length cannot enforce that limit. Test a fixture whose encoded size fits the budget but whose decoded size exceeds it, along with a positive case that remains within the budget.

async function collectWithin(response, max) {
  const reader = response.body.getReader();
  const chunks = [];
  let size = 0;
  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) return Buffer.concat(chunks, size);
      size += value.byteLength;
      if (size > max) {
        await reader.cancel();
        throw new RangeError('decoded body exceeds budget');
      }
      chunks.push(Buffer.from(value));
    }
  } finally {
    reader.releaseLock();
  }
}

assert.ok(zipped.length < 100 && original.length > 100);
await assert.rejects(collectWithin(await fetch(url), 100), RangeError);
assert.deepEqual(await collectWithin(await fetch(url), original.length), original);
await new Promise(resolve => upstream.close(resolve));

This budget limits what the sample collector retains, not how much work the decoder has already done when a chunk reaches the application. It is also separate from a limit on encoded bytes and from a request deadline. A gateway with a requirement to bound both representations should test both and confirm which layer can actually observe each count. Do not infer a decoded body budget from an upstream header supplied by another service.

The fixture buffers the accepted bytes to keep the result easy to verify. For a route that streams downstream, plan what to do if the limit is crossed after some bytes have already been sent. If consumers need an all or nothing result, check the complete candidate before committing the response. For related byte versus text decisions, see the JSON binary encoding tests.

Frequently Asked Questions

Does Node.js Fetch decompress gzip responses automatically?
The built in Fetch implementation tested here decodes a response labeled with gzip before your code consumes its body. Inspect the result with a real local HTTP response. A Response object that you construct yourself with arbitrary bytes and headers is not a test of the network client's decoding behavior.
Why does the Fetch body length differ from Content-Length?
The upstream header can describe compressed bytes, while the body exposed by Fetch contains decoded bytes. Those lengths can differ even for a completely successful response. Compare lengths only within the same representation, and compute or omit the length header when you send a new body downstream.
Can I forward Content-Encoding from Fetch to a browser?
Forward a gzip encoding label only if the outgoing bytes are actually gzip encoded. When Fetch has already decoded the body and the proxy sends those decoded bytes, remove the stale label. Otherwise the next client may attempt to decompress ordinary JSON and fail before it can parse the document.
How can a Node.js proxy keep the original gzip bytes?
Use an HTTP client interface that exposes the encoded response bytes, and verify that behavior with a fixture before forwarding them. The Node HTTP client used in this example exposes those bytes. Preserve the matching encoding header only when the bytes are unchanged, and test what a downstream client receives.
Does Response.json limit the size of decompressed JSON?
Do not use Response.json as your application size check. It consumes and parses the body before returning the result. If your policy limits decoded bytes, read the stream with a counter and cancel when the limit is exceeded. This application limit does not replace transport timeouts or other resource controls.

Working with JSON right now?

Paste it into JSON Prettifier to format, validate, or minify it. Everything runs in your browser.

Open JSON Prettifier