Traces¶

When to Use¶

A trace follows one operation, such as a request or job, through a set of related spans. Each span records the start and end of one part of the work. Parent-child relationships show which operation started another; links describe other causal relationships.

Use traces to locate latency, separate application work from dependency calls, and investigate a particular failure across services. Use metrics for rates and alerts across many operations, and logs for details that must remain available without an active span.

Create spans for meaningful operations, not every function or loop iteration. Keep names stable, such as checkout.price, rather than including customer IDs or raw URLs. Let existing HTTP and database integrations create their standard spans.

Spans, Attributes, Events, and Links¶

The span's kind describes its role: INTERNAL for business work, SERVER for an incoming request, CLIENT for an outgoing call, and PRODUCER or CONSUMER for sending or receiving messages. Follow the integration's semantic conventions when choosing kinds and standard attributes.

A span has at most one parent. For example, work combining several messages may need links to each source rather than treating one source as its only cause. Keep events sparse and limit their attributes; a span's event list is not a replacement for high-volume logs.

Python Pattern¶

After SDK setup, obtain a tracer from the API. This wrapper records the elapsed time of an existing async price calculation:

from collections.abc import Awaitable, Callable

from opentelemetry import trace

tracer = trace.get_tracer("checkout.telemetry")


async def price_order(calculate_order_total: Callable[[], Awaitable[float]]) -> float:
    with tracer.start_as_current_span("checkout.price") as span:
        span.set_attribute("checkout.pricing.strategy", "standard")
        order_total = await calculate_order_total()
        span.add_event("checkout.price.calculated")
        return order_total

The ordinary with block keeps the span active across the await and ends it on exit. Its duration includes waiting; it is not a measurement of CPU time.

Errors and Existing Spans¶

By default, an exception escaping start_as_current_span is recorded and sets the span's status to ERROR. Do not record the same exception manually before re-raising it. Exception records may contain messages and stack traces with sensitive information; disable automatic recording where necessary and record only sanitized details.

If code catches an exception inside the span but the operation still fails, explicitly record the permitted exception details and set Status(StatusCode.ERROR). Recording an exception alone does not set status. A retry that eventually succeeds does not necessarily make the enclosing operation a failure; record limited retry details without marking successful requests as errors.

To add attributes to a framework-created span, use trace.get_current_span(). Check span.is_recording() before expensive telemetry-only computation. Business work and metric recording must still run when the span is not recording.

Context Propagation¶

The active context identifies the current span. New spans normally use it to select their parent. Within normal Python async execution, this context follows context-local state. It does not cross a network or process boundary by itself.

Propagation serializes context into a carrier, such as HTTP headers or message metadata. The sender injects context into that carrier; the receiver extracts it and uses it when starting its span. Instrumented clients and servers normally handle this themselves. Check context handling separately for the actual thread executor or queue integration.

The following example illustrates injection and extraction using a string-keyed dictionary. It does not send a real message:

from opentelemetry import propagate, trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("checkout.telemetry")
outgoing_headers: dict[str, str] = {}
with tracer.start_as_current_span("checkout.send", kind=SpanKind.PRODUCER):
    propagate.inject(outgoing_headers)

received_headers = outgoing_headers.copy()
parent_context = propagate.extract(received_headers)
with tracer.start_as_current_span(
    "checkout.consume", context=parent_context, kind=SpanKind.CONSUMER
):
    pass

In real code, send the message while the producer span is active and extract context from the received message. Custom getters and setters adapt propagation to carriers other than string-keyed dictionaries, including headers with multiple values.

For batches or delayed work, use the messaging conventions to decide between a parent relationship and links. Not every message delivery should be a direct child of the sending span.

Sampling and Cost¶

Sampling selects which traces to record and export. Head sampling decides when a span starts. A parent-based policy follows the parent's sampling decision; for a span without a parent, it applies a root policy.

For explicit SDK setup, ParentBased(TraceIdRatioBased(0.1)) illustrates retaining about 10% of new root traces and following that decision for their descendants. Configure the sampler on TracerProvider. Choose the rate for the application's traffic, cost, and diagnostic needs; 10% is not a general default.

Head sampling cannot guarantee retention of failures discovered later. Tail sampling in a Collector can select traces using later evidence, such as errors or long durations. The required spans must reach that Collector, and related spans must be routed to the same sampling decision point. Plan memory and processing capacity accordingly. Tail sampling cannot recover spans already dropped by head sampling.

Trace sampling does not automatically sample Python logs or metric measurements. A metric intended to count every request must not be derived only from sampled spans. At public boundaries, decide whether to honor incoming sampling flags and which context may be sent to external services.

What to Verify¶

Use a provider local to the test and InMemorySpanExporter to inspect completed spans. Check names, parents or links, kinds, event limits, error status, and absence of secrets.

An integration check across a sender and receiver should show the same trace ID and the intended parent relationship or links. Include concurrent requests to detect context leaking between operations, and verify sampling for both root and child spans.