Building Distributed Tracing and Observability with AWS X-Ray

Introduction

Modern applications rarely live in a single process or a single repository. A typical user action may traverse a web client, an authentication layer, a business API, an event bus, and one or more serverless workers. When something goes wrong or when latency spikes, engineers need a way to follow one request across all of those hops without manually stitching together timestamps and guesswork.

This article describes a pragmatic observability pattern we implemented across a distributed stack: a per-request correlation identifier, AWS X-Ray segments at each tier, structured logging keyed on the same identifier, and optional session replay linkage for front-end debugging. The approach is intentionally logs-first and incrementally adoptable: you can ship correlation IDs and searchable logs before every service emits perfect traces.

The Problem with Siloed Observability

Each layer of a distributed system usually has its own logs, metrics, and if you are lucky traces. Without a shared key, investigation looks like this:

• The web team searches browser network headers and console output.
• The API team greps application logs by user ID or timestamp.
• The serverless team inspects CloudWatch for a narrow time window.
• Nobody agrees on whether the failure happened before or after the event was published.

AWS X-Ray helps by visualizing service maps and trace timelines, but X-Ray alone does not automatically unify arbitrary microservices unless they participate in the same trace context or share searchable metadata. That gap is where a correlation ID becomes the contract that ties everything together.

What Are Traces and Segments?

In AWS X-Ray, a trace represents the end-to-end path of a request. A segment represents work performed by one component often one service handling one inbound call. Segments can contain subsegments for downstream calls such as HTTP requests, database queries, or SDK operations.

Annotations are indexed key-value pairs on a segment. They are ideal for correlation IDs because you can filter traces in the X-Ray console with expressions like:

annotation.correlation_id = "<uuid>"

Metadata, by contrast, is useful for debugging detail but is not indexed for the same fast search experience.

The Correlation ID Contract

We standardized on a single HTTP header for synchronous paths:

x-correlation-id

The value is a UUID generated once per user-initiated request at the edge (the web client). Every downstream service is responsible for:

• Reading the header if present
• Generating a UUID only when the header is missing (e.g., background jobs)
• Echoing the value in responses where appropriate
• Forwarding the header on outbound HTTP calls
• Stamping the same value onto asynchronous event payloads

For asynchronous flows, we carry the same UUID inside event metadata for example, detail.metadata.correlationId on EventBridge-style envelopes so serverless consumers do not mint a new random identifier and break the chain.

We deliberately kept an existing reference or request ID header unchanged. Correlation ID is additive: it is the cross-service search key, not a replacement for legacy tracing fields.

Reference Architecture

The following pattern applies generically to a browser application, an authentication/API edge, a core backend API, and event-driven workers:

Two trace graphs are normal: synchronous HTTP often shares one AWS trace ID via X-Amzn-Trace-Id, while async EventBridge → Lambda typically starts a new trace. The correlation ID is what joins those graphs in search tools.

Implementation by Tier

1. Web Client

An HTTP interceptor runs on every outbound request. It ensures x-correlation-id is set before the call leaves the browser. On failures, the client logs the correlation ID alongside method, URL, and status.

We intentionally did not instrument the browser with the X-Ray SDK. Front-end tracing adds complexity and rarely matches backend trace IDs anyway. Instead, the client propagates the UUID and, for support workflows, can emit analytics events that bind the correlation ID to a session replay identifier when errors occur.

2. Edge API (Node.js / Express)

Middleware runs early in the request pipeline:

Let's Build Something Great Together

Ready to transform your idea into a powerful software solution? Talk to our experts and get a free consultation.

Contact Us

• Resolve or generate the correlation ID from inbound headers
• Store it on the request object and in async-local storage for loggers
• Open an X-Ray segment (via aws-xray-sdk) and add annotation correlation_id
• Set the response header so clients can read it back
• Include the header in CORS allow/expose lists

Outbound proxy helpers map the correlation header whenever the edge service calls the core API so the backend sees the same UUID.

On Elastic Beanstalk, enable the X-Ray daemon (XRayEnabled: true) and attach the AWSXRayDaemonWriteAccess managed policy to the instance profile.

3. Core Backend API

A bootstrap component hooks application lifecycle events:

• Before request: read x-correlation-id and x-amzn-trace-id
• Begin an X-Ray segment (we used pkerrigan/xray with ext-sockets)
• Annotate correlation_id on the segment
• After request: set HTTP status and submit the segment to the daemon

Logging targets append the correlation ID as a dedicated field so grep and CloudWatch Logs Insights queries stay simple.

Infrastructure requirements: PHP sockets extension, X-Ray daemon on the instance, and IAM permissions to put trace segments.

4. Serverless Workers (AWS Lambda)

Infrastructure-as-code enables Active Tracing on every function and attaches AWSXRayDaemonWriteAccess to execution roles. That gives you baseline segments without modifying every handler.

Application code adds a shared observability helper that:

• Extracts correlationId from API Gateway headers, EventBridge detail, or SQS-wrapped bodies
• Logs a single JSON line: {"correlationId":"...","handler":"..."}
• Annotates X-Ray with correlation_id

Important Lambda detail: Active Tracing exposes a facade segment that your handler code cannot annotate directly. The fix is to create a short subsegment, attach annotations there, close the subsegment, and restore the facade. Without that subsegment, X-Ray filters on annotation.correlation_id will miss Lambda even when CloudWatch logs contain the UUID.

How Engineers Search Incidents

X-Ray

Filter traces with:

annotation.correlation_id = "<paste-uuid-here>"

Expect multiple trace results for one user action when async workers are involved. That is normal.

CloudWatch Logs Insights

Run across auth, API, and Lambda log groups:

fields @timestamp, @message
| filter @message like /<correlation-id>/
| sort @timestamp asc
| limit 200

Session Replay (Optional)

When product analytics with session recording is enabled, HTTP error paths can emit events containing both correlationId and the analytics session identifier. Support teams filter by correlation ID in the analytics product and jump to the recording of what the user saw without exposing internal service names in customer-facing tooling.

Design Decisions and Trade-offs

• Logs-first. Correlation IDs and structured logs deliver value immediately. X-Ray annotations enhance search but require per-runtime instrumentation.
• Single header, lowercase. HTTP headers are case-insensitive, but we standardize on x-correlation-id everywhere to avoid defensive parsing sprawl.
• No browser X-Ray. The client propagates IDs only; backend systems own trace segments.
• Async is a separate trace. Do not force one X-Ray tree. Use correlation ID as the join key.
• Incremental IaC. Enable Lambda Active Tracing platform-wide first; add handler annotations second.

Conclusion

Distributed tracing is not only about colorful service maps it is about giving every engineer the same search key for one user action. AWS X-Ray segments show where time was spent; correlation IDs show which logs and traces belong together even when AWS assigns different trace IDs to synchronous and asynchronous legs.

Start with a UUID at the edge, propagate it religiously, log it everywhere, and annotate X-Ray segments where your runtime allows. That combination turns multi-service debugging from a scavenger hunt into a filtered query and that is observability that pays for itself the first time production misbehaves on a Friday afternoon.