Structured Logging With JSON: Why Your Logs Should Be Machine-Readable

Quick answer: Structured logging means writing each log event as a JSON object with named fields (timestamp, level, message, service, trace_id, plus whatever context matters) instead of a free-text line. Write one object per line to stdout, let a log shipper collect it, and you can filter, group, and alert on any field in Datadog, Loki, Elasticsearch, or CloudWatch without writing a single regular expression.

It is 2 a.m. and checkout is failing for some customers. The log line says Payment retry for user 8812 attempt 3 failed: timeout. To find out how many users are affected you need a regex that pulls out the user ID, hope every developer formatted the message the same way, and discover that half the lines say attempt=3 and the other half say attempt 3. With structured logs the question is one filter: level:error AND message:"payment retry", grouped by user_id. That is the whole argument.

Inspect complete individual records in the JSON Viewer, or use the JSONPath Query Tester to select fields from a parsed record. Neither tool replaces an incremental stream parser.

Plain Text vs JSON Logs

2026-10-08 14:32:07 WARN  Payment retry for user 8812 attempt 3 failed: timeout after 5000ms

{"timestamp":"2026-10-08T14:32:07.412Z","level":"warn","message":"payment retry failed","service":"checkout-api","env":"prod","user_id":8812,"attempt":3,"reason":"timeout","timeout_ms":5000,"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

The first line is pleasant to read and nearly useless to query. The second is unpleasant to read raw and trivial to query: user_id is a number you can count distinct values of, timeout_ms is a number you can average, and trace_id links this event to every other service that handled the same request. Readability for humans is a development-time concern, solved by a pretty-printer. Queryability is a production concern, solved by structure.

Standard Fields Every Log Event Should Carry

Three of these deserve emphasis. The timestamp must include a timezone, and should be UTC; a bare 2026-10-08 14:32:07 is ambiguous the moment logs from two regions are merged. See JSON date formats for why ISO 8601 with a Z suffix is the only sane choice. The message should be a short constant string with no interpolated values; "payment retry failed" is groupable, "payment retry for user 8812 failed" is 8,812 different messages. And trace_id is what turns logs from a per-service pile into a cross-service story; if you run OpenTelemetry, inject it from the active span automatically.

Exact key names vary by ecosystem. Elastic Common Schema uses @timestamp, log.level, and trace.id; OpenTelemetry uses trace_id; Datadog recognizes status and message. Pick one convention, write it down, and make the logger enforce it so no one has to remember. The key naming conventions guide has the general rules.

Setting Up a JSON Logger

Node.js: pino

import pino from "pino";

const logger = pino({
  level: process.env.LOG_LEVEL || "info",
  base: { service: "checkout-api", env: process.env.NODE_ENV },
  timestamp: pino.stdTimeFunctions.isoTime,
  formatters: { level: (label) => ({ level: label }) },
  redact: ["req.headers.authorization", "*.password", "card.number"],
});

logger.warn({ user_id: 8812, attempt: 3, reason: "timeout" }, "payment retry failed");
// {"level":"warn","time":"2026-10-08T14:32:07.412Z","service":"checkout-api","env":"prod","user_id":8812,"attempt":3,"reason":"timeout","msg":"payment retry failed"}

pino's defaults are numeric levels (30 for info) and epoch milliseconds, which are fast but less friendly to humans and some platforms; the formatters and timestamp options above switch to labels and ISO strings. The redact option is not optional in practice; more on that below. In development, pipe through pino-pretty; in production, never.

Python: structlog

import structlog

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.JSONRenderer(),
    ]
)

log = structlog.get_logger(service="checkout-api")
log.warning("payment retry failed", user_id=8812, attempt=3, reason="timeout")
# {"service": "checkout-api", "user_id": 8812, "attempt": 3, "reason": "timeout", "event": "payment retry failed", "level": "warning", "timestamp": "2026-10-08T14:32:07.412Z"}

If you must stay on the standard logging module, python-json-logger provides a JsonFormatter you can attach to any handler, and the extra= argument carries the fields. structlog's bind() and contextvars are what make request-scoped fields like trace_id appear on every line without passing them around.

Go: zerolog or zap

// zerolog
logger := zerolog.New(os.Stdout).With().Timestamp().Str("service", "checkout-api").Logger()
logger.Warn().Int("user_id", 8812).Int("attempt", 3).Str("reason", "timeout").Msg("payment retry failed")

// zap
logger, _ := zap.NewProduction()
logger.Warn("payment retry failed", zap.Int("user_id", 8812), zap.Int("attempt", 3), zap.String("reason", "timeout"))

Both are zero-allocation JSON loggers and both default to JSON output. zerolog's default timestamp format is RFC 3339 under a time key; zap's production config writes epoch seconds under ts, which you can change through EncoderConfig. Go 1.21 also added log/slog to the standard library with a built-in JSONHandler, which is enough for many services.

Java: Logback with the Logstash encoder

<!-- logback.xml -->
<configuration>
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <customFields>{"service":"checkout-api"}</customFields>
    </encoder>
  </appender>
  <root level="INFO"><appender-ref ref="STDOUT"/></root>
</configuration>

import static net.logstash.logback.argument.StructuredArguments.kv;

log.warn("payment retry failed", kv("user_id", 8812), kv("attempt", 3), kv("reason", "timeout"));

logstash-logback-encoder is the de facto JSON encoder for Logback and works with SLF4J unchanged. MDC values are included automatically, which is the usual place to put trace_id. Log4j 2 has an equivalent JsonTemplateLayout.

The File Format Is JSON Lines, Not a JSON Array

A log file is not [ {...}, {...} ]. It is one complete JSON object per line, newline-terminated, with no outer array and no commas between records. That format is called JSON Lines or NDJSON, and it is the only shape that can be appended to atomically, tailed with tail -f, rotated, and partially read after a crash. Every logger above produces it by default; every shipper (Fluent Bit, Vector, Promtail, the Datadog agent, the CloudWatch agent) expects it.

One consequence: a .log file of JSON Lines is not itself valid JSON, so a whole-file validator rejects it. To inspect a single suspicious event, copy that one line into JSON Prettifier, which formats it with indentation and shows you nested fields you would otherwise miss. Write to stdout, not to files; the twelve-factor convention is that the platform captures stdout and decides where it goes.

Querying JSON Logs

Locally with jq

# Only errors
jq -c 'select(.level == "error")' app.log

# Everything for one trace, pretty
jq 'select(.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")' app.log

# Count events by message
jq -r '.message' app.log | sort | uniq -c | sort -rn | head

# A readable table: time, level, message
jq -r '[.timestamp, .level, .message] | @tsv' app.log

# Average timeout_ms across warnings
jq -s '[.[] | select(.level == "warn") | .timeout_ms] | add / length' app.log

jq reads JSON Lines natively, one value at a time, so there is no need to wrap the file. The jq cheat sheet has forty more of these.

In a log platform

# CloudWatch Logs Insights
fields @timestamp, message, user_id, attempt
| filter level = "error" and service = "checkout-api"
| stats count(*) by user_id
| sort count desc

# Grafana Loki (LogQL)
{service="checkout-api"} | json | level="error" | attempt >= 3

# Datadog log search
service:checkout-api status:error @attempt:>=3

All three parse the JSON on ingest and turn each key into a queryable field automatically. That is the payoff: the attempt field you added in a two-line commit is filterable and graphable the moment the first line arrives, with no parser configuration. Note that each platform has its own reserved names for level and message (Datadog uses status, Loki uses whatever you extract, CloudWatch treats everything as a field), so check the mapping once and configure the logger to match.

Pitfalls and How to Avoid Them

• Logging secrets. logger.info({ req }) writes the Authorization header, session cookies, and whatever was in the body. Use the logger's redaction (pino redact, structlog processors, a Logback MaskingJsonGeneratorDecorator) and log request summaries (method, path, status, duration), never raw request objects. The JSON security guide covers the rest of the surface.
• Huge nested objects. A full ORM entity with its relations, or an entire API response, produces a 40 KB line that costs money to ingest and store and gets truncated by many shippers, which cap line length at a limit that varies by agent. Log identifiers and counts, not payloads.
• Interpolating values into the message. "Retry " + n + " failed" destroys groupability. Keep the message constant and put n in a field.
• Inconsistent key names. userId in one service, user_id in another, uid in a third means three filters for one question. Define the shared keys in one place and have each logger apply them as base fields.
• Type drift. "attempt": 3 today and "attempt": "3" tomorrow breaks numeric queries and, in Elasticsearch, can reject the document outright because the field was mapped as a number.
• Non-UTF-8 bytes. Logging a raw Buffer or a filename from a legacy system can produce invalid UTF-8 that the shipper drops silently. Decode or base64-encode binary before logging. The encoding guide shows what the failures look like.
• Multi-line stack traces. A stack trace printed with real newlines becomes twenty separate "log lines" in the platform. Put it in an error.stack string field, where the \n characters are escaped and the event stays on one line.
• Double-encoded JSON. If a field contains "{\"a\":1}" with escaped quotes, something called JSON.stringify before handing the object to the logger. Pass objects, not strings.
• Local-time timestamps. 2026-10-08T14:32:07 with no zone is interpreted differently by every consumer. Always emit UTC with a Z or an explicit offset.

A Checklist Before You Ship

1. Every line is one valid JSON object, UTF-8, newline-terminated.
2. Timestamp is ISO 8601, UTC, with milliseconds.
3. level, message, service, env are present on every event.
4. trace_id or request_id is injected automatically, not by hand.
5. Messages are constant strings; variable data lives in fields.
6. Redaction is configured and tested with a fake token.
7. Development uses a pretty-printer; production writes raw JSON to stdout.
8. The key naming convention is documented and matches your log platform's reserved fields.