Tempo: Distributed Tracing with Grafana

By Khimananda Oli | Last reviewed: August 2026

Debugging latency in microservices often fails because traditional tracing backends cannot handle modern data volumes without massive index overhead. Tempo: Distributed Tracing with Grafana solves this by decoupling trace storage from indexing, using cheap object storage like S3 or GCS instead of expensive block storage. If you are already running a Grafana-centric observability stack, adopting Tempo eliminates vendor lock-in and reduces operational complexity significantly. This guide covers the architecture, configuration, and production hardening required to make it work reliably.

How does Tempo: Distributed Tracing with Grafana architecture differ from Jaeger?

Understanding why Tempo exists requires looking at the limitations of first-generation tracing systems. Traditional backends like Jaeger or Zipkin typically depend on Elasticsearch or Cassandra for both storage and indexing. While functional, these databases become operationally heavy and expensive as trace ingestion crosses gigabytes per hour. You end up managing complex database clusters just to store ephemeral debug data.

Tempo takes a fundamentally different approach inspired by log aggregation tools like Loki. It treats traces as opaque blobs stored in object storage, maintaining only a lightweight index of trace IDs locally. This separation means your storage costs scale linearly with commodity cloud pricing rather than premium database IOPS. For teams in Nepal or emerging markets where cloud budget efficiency is critical, or global teams managing petabyte-scale telemetry, this distinction matters immensely. As discussed in observability vs monitoring fundamentals, choosing the right storage tier for each signal type is the foundation of sustainable platform engineering.

The architecture consists of five distinct microservices, though you can run them as a single binary for smaller deployments. The Distributor receives spans and hashes them by trace ID to ensure all spans for a single request hit the same Ingester. The Ingester batches incoming traces in memory and writes them to a Write-Ahead Log (WAL) before flushing complete blocks to object storage. The Compactor runs asynchronously to merge small blocks into larger ones and maintain the bloom filters used for lookup. Finally, the Query Frontend and Querier handle retrieval by scanning the index and fetching specific byte ranges from storage. This design allows you to scale ingestion and querying independently, a crucial capability when traffic spikes unpredictably.

How do you configure Tempo with OpenTelemetry and object storage?

Configuration is where many engineers stumble because Tempo requires explicit definitions for storage backends and receiver protocols. Unlike monolithic tools that auto-detect everything, Tempo expects a declarative YAML configuration. Below is a production-grade starting point for a Kubernetes deployment using S3-compatible storage and OpenTelemetry protocol.

server:
  http_listen_port: 3200
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"
    jaeger:
      protocols:
        grpc:
          endpoint: "0.0.0.0:14250"

ingester:
  max_block_duration: 5m
  lifecycler:
    ring:
      replication_factor: 2

compactor:
  compaction:
    block_retention: 48h
  sharding_ring:
    replication_factor: 2

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-prod
      endpoint: s3.amazonaws.com
      region: us-east-1
      access_key: "${AWS_ACCESS_KEY_ID}"
      secret_key: "${AWS_SECRET_ACCESS_KEY}"
      insecure: false
    wal:
      path: /var/tempo/wal
    cache:
      caches:
        - memcached:
            host: memcached.tracing.svc.cluster.local
            service: memcached-client
            addresses: []

Several details in this configuration warrant attention. First, always set max_block_duration in the ingester to control flush frequency; 5 minutes balances memory usage against object storage API calls. Second, never hardcode credentials in the config file. Use environment variable substitution or integrate with secrets management solutions to inject AWS keys securely. Third, enable caching immediately. Without Memcached or Redis in front of object storage, every query triggers multiple GET requests, making the UI unusable during incident response. The WAL path must be on persistent storage; losing the WAL means losing unflushed traces during pod restarts.

Validating your ingestion pipeline

After deploying, verify that spans are actually reaching storage before trusting the system. Use the Tempo API directly to check build info and readiness:

1. Check cluster health: curl http://tempo:3200/ready should return "ready".
2. Verify receiver status: Query the metrics endpoint /metrics and search for tempo_distributor_spans_received_total. A flat line indicates misconfigured receivers or network policies blocking OTLP ports.
3. Test trace retrieval: Send a known test trace via telemetrygen, then query GET /api/traces/{traceID} to confirm round-trip integrity.

Why should you choose Tempo over Jaeger or Elasticsearch for tracing?

Selecting a tracing backend involves trade-offs between cost, operational burden, and query flexibility. Teams migrating from legacy stacks often ask whether the switch is justified. The answer depends heavily on your scale and existing ecosystem. If you are already standardized on Grafana for metrics and logs, Tempo provides native correlation features that standalone tools cannot match without significant glue code.

The primary advantage of Tempo is economic and operational. Object storage is roughly 10x cheaper than provisioned IOPS SSDs required by Elasticsearch. More importantly, you stop babysitting Java heap sizes and shard allocation. The trade-off is query capability: Tempo is optimized for "find this specific trace" or "show recent errors," not arbitrary full-text analytics across billions of spans. For most debugging workflows, this limitation is acceptable. If you need deep analytical queries regularly, consider exporting sampled data to ClickHouse or keeping a hot tier in Elasticsearch while archiving everything else in Tempo.

How do you optimize Tempo performance and manage retention in production?

Deploying Tempo is straightforward; keeping it performant under load requires tuning. The most common failure mode in production is overwhelming the queriers during incident investigations when multiple engineers search simultaneously. Since Tempo lacks a comprehensive inverted index, queries scan metadata files in object storage. Caching is mandatory, not optional.

Tuning caching and compaction

Configure Memcached or Redis specifically for three distinct caches: Bloom filters, trace indexes, and footer offsets. Bloom filter caching prevents unnecessary object storage reads for non-existent trace IDs. Footer caching accelerates parquet file parsing. In my experience helping teams pass SOC 2 audits, demonstrating controlled data retention and reliable retrieval performance is often part of the evidence collection process. Proper compaction settings directly support this by ensuring old data is predictably archived or deleted according to policy.

Set compaction windows aggressively during off-peak hours. The default compaction cycle may leave too many small blocks during high-ingestion periods, increasing read amplification. Adjust compaction_window to 1h or 2h depending on your throughput. Monitor tempo_compactor_blocks_compacted_total versus tempo_query_frontend_queries_total to identify bottlenecks. If query latency exceeds 2 seconds consistently, increase querier replicas or expand cache capacity before scaling storage.

Managing data lifecycle and compliance

Retention in Tempo is governed entirely by the compactor's block_retention setting. Unlike databases with TTL per document, Tempo deletes entire blocks based on creation time. This makes retention predictable but granular only to the block level. For compliance frameworks requiring specific retention periods (e.g., 90 days for financial transactions), align your block retention with regulatory requirements. Remember that deleting from object storage is eventually consistent; audit logs should track compactor deletion events separately for verification. Integrating this with automated compliance pipelines ensures retention policies remain enforced even as infrastructure changes.

Implementing Tempo: Distributed Tracing with Grafana for Production Reliability

Adopting Tempo: Distributed Tracing with Grafana transforms observability from a cost center into a scalable engineering enabler. Success depends on respecting its architectural constraints: embrace object storage, invest in caching, and align retention with business needs rather than technical defaults. Start with a modest deployment, validate your OpenTelemetry instrumentation thoroughly, and monitor compaction metrics before scaling to full production traffic. The initial setup effort pays dividends in reduced operational toil and predictable cloud bills.

If your team needs guidance implementing distributed tracing, optimizing existing observability stacks, or preparing infrastructure for compliance audits, reach out to discuss your specific requirements. Practical, battle-tested architecture advice beats generic documentation every time.