Apache Pulsar Overview

By Khimananda Oli | Last reviewed: August 2026

Choosing the right event streaming platform is a high-stakes architectural decision that dictates your operational overhead for years. This Apache Pulsar overview cuts through marketing claims to explain the storage-compute separation model that fundamentally changes how you scale, retain data, and manage multi-tenant workloads compared to legacy brokers. If you are evaluating streaming infrastructure for complex event processing or long-term retention, understanding these mechanical differences is essential before committing to a cluster.

How does Apache Pulsar architecture differ from traditional message brokers?

The defining characteristic of Pulsar is the strict decoupling of the serving layer from the storage layer. In traditional log-based brokers like Kafka, the broker process owns the local disk; if a broker fails, partition leadership must transfer to another broker that may not have the data locally, triggering expensive replication or recovery. Pulsar eliminates this coupling by using Apache BookKeeper as a dedicated, segmented storage backend. Brokers become stateless proxies that handle protocol translation, authentication, and dispatching, while Bookies (storage nodes) handle durability via quorum writes.

This separation has profound operational implications. When you need to scale ingestion throughput, you add brokers without moving any existing data. When you need more storage capacity or IOPS, you add Bookies, and new ledger segments automatically distribute across them without rebalancing old data. For teams managing event-driven microservices, this means maintenance windows no longer require massive data shuffling. The metadata about which Bookie holds which segment lives in ZooKeeper or Oxia, allowing any available broker to serve reads for any topic instantly.

Understanding Ledgers and Segments

Unlike a single monolithic log file per partition, Pulsar topics are composed of a sequence of ledgers. Each ledger is an append-only log replicated across a configurable ensemble of Bookies. When a ledger reaches a size or time threshold, it is sealed and a new one is created. This segmentation is what makes instant failover possible: if a Bookie dies, only the active ledger needs to be recovered onto a replacement node, not terabytes of historical data. Sealed ledgers are immutable and can be safely offloaded to object storage, forming the basis of tiered storage.

What is tiered storage and why does it matter for cost optimization?

Tiered storage is arguably Pulsar’s most practical feature for production economics. It allows you to configure policies where sealed ledgers are automatically uploaded to cheap object storage (S3, GCS, Azure Blob) after a defined retention period or size threshold. Crucially, the data remains accessible through the same Pulsar consumer API; the broker transparently fetches segments from object storage when a consumer requests older offsets. This turns Pulsar into a unified system for both real-time messaging and long-term event archival.

In practice, this eliminates the need for separate ETL pipelines that drain hot broker disks into a data lake for compliance or replay. For fintech companies in Nepal subject to data retention regulations, or global SaaS platforms needing GDPR-compliant audit trails, you can retain months or years of events at S3 pricing while keeping the last few hours on fast NVMe for low-latency consumption. The offloading process is asynchronous and does not impact ingest performance.

# Example: Configure tiered storage policy via pulsar-admin
pulsar-admin namespaces set-offload-policies \
  --offloadAfterElapsed 3600 \
  --offloadAfterSizeInBytes 1073741824 \
  --offloadStorageType aws-s3 \
  --s3ManagedLedgerOffloadBucket my-pulsar-archive \
  --s3ManagedLedgerOffloadRegion ap-south-1 \
  tenant/namespace

# Verify offload status
pulsar-admin topics stats persistent://tenant/namespace/topic-name

A common mistake is setting the offload threshold too aggressively. Offloading very small ledgers creates excessive API calls to object storage. I typically recommend a minimum segment size of 256MB–1GB and a time threshold of at least one hour to balance cost against retrieval latency. Remember that reading from tiered storage adds network hops; reserve hot storage for your SLA-critical recent window.

How do you implement multi-tenancy and isolation in Apache Pulsar?

Pulsar implements multi-tenancy natively through a hierarchical namespace model: tenant/namespace/topic. Tenants map to organizational units or customers, namespaces group related topics, and topics are the actual message channels. Unlike systems where isolation requires separate clusters or complex ACL hacks, Pulsar enforces resource quotas, authentication, and authorization boundaries at each level of this hierarchy out of the box.

You can enforce publish and subscribe rate limits, backlog quotas, and encryption requirements per namespace. This is critical for platform teams serving multiple product squads or external customers. Instead of provisioning and maintaining three separate clusters for dev, staging, and prod—or worse, mixing them insecurely—you run one cluster with hard isolation guarantees. Authentication integrates with OIDC, mTLS, or token-based providers, and authorization policies are granular down to individual topics.

Practical Isolation Configuration

• Backlog Quotas: Prevent runaway producers from filling storage. Set backlogQuotaLimitSize and choose a policy (producer_exception, consumer_backpressure, or discard).
• Rate Limiting: Apply publishRate and subscribeRate at the namespace level to protect shared Bookies from noisy neighbors.
• Encryption: Enforce end-to-end encryption per namespace so even cluster admins cannot read sensitive payloads without keys.
• Schema Validation: Enable schema compatibility checks per namespace to prevent breaking changes in shared event contracts.

When should you choose Apache Pulsar over Apache Kafka?

This is the most common question I encounter during architecture reviews. Both are excellent, but they optimize for different trade-offs. Pulsar’s advantages shine in specific scenarios: when you need true independent scaling of compute and storage, when long-term retention must be cost-effective without external pipelines, when native multi-tenancy is non-negotiable, or when you require flexible subscription modes (exclusive, shared, failover, key-shared) on the same topic. Kafka often wins on raw single-topic throughput simplicity, ecosystem maturity for stream processing (Kafka Streams, ksqlDB), and operational familiarity among hiring pools.

If your primary use case is high-throughput log aggregation or stream processing with a team already skilled in Kafka, stay with Kafka. If you are building a multi-tenant platform, need to retain petabytes of events cheaply, or require diverse consumption patterns (e.g., queue-like semantics alongside pub/sub), Pulsar’s architectural bets pay off. For teams observing these systems, integrating Prometheus and Grafana monitoring is straightforward for both, though Pulsar exposes richer per-subscription metrics natively.

How do you deploy and operate Apache Pulsar in production?

Production Pulsar deployments demand discipline. The canonical deployment target in 2026 is Kubernetes via the official Helm chart or the Pulsar Operator. You must treat ZooKeeper (or Oxia) as the most critical stateful component; its loss means cluster metadata loss. Always deploy ZK in odd numbers (3 or 5) across failure domains with dedicated PVs. Bookies should use dedicated disks (NVMe preferred) and never share I/O with other workloads. Brokers are stateless and can run as Deployments with HPA, but ensure JVM heap and direct memory are tuned separately—direct memory exhaustion is the #1 cause of broker OOM kills.

Monitoring is non-optional. Expose JMX metrics via the Prometheus JMX exporter and track these golden signals: broker message rate, BookKeeper write latency (p99), ledger under-replication count, and subscription backlog growth. Set up alerts on backlog growth rate, not absolute size, to catch consumer stalls early. For teams new to defining reliability targets, start by establishing meaningful SLIs and SLOs around publish latency and consumer lag before tuning aggressive autoscaling.

Essential Production Tuning Steps

1. Separate Journal and Ledger Disks: BookKeeper writes journals sequentially and ledgers randomly. Mixing them on the same disk kills throughput. Use separate PVCs or physical disks.
2. Tune Direct Memory: Set -XX:MaxDirectMemorySize explicitly. Default JVM settings often allocate too little for Pulsar’s Netty buffers, causing silent performance degradation before OOM.
3. Enable Schema Registry: Turn on schema validation early. Retroactive enforcement breaks producers. Use AVRO or Protobuf with backward compatibility checks.
4. Configure Graceful Shutdown: Set brokerShutdownTimeoutMs and pre-stop hooks in K8s to allow in-flight messages to drain. Abrupt pod kills cause duplicate deliveries.
5. Test Failure Modes: Regularly kill Bookies and brokers in staging. Verify auto-recovery completes within your RTO. Practice ledger recovery drills quarterly.

Evaluating Apache Pulsar for Your Next Streaming Platform

This Apache Pulsar overview has covered the architectural foundations that make it distinct: storage-compute separation, tiered storage economics, native multi-tenancy, and production operational realities. Pulsar is not a drop-in Kafka replacement; it is a deliberate choice for teams willing to invest in understanding its primitives in exchange for superior flexibility at scale. If your requirements include multi-year retention, isolated multi-team workloads, or dynamic scaling without data migration pain, Pulsar deserves serious evaluation.

Before spinning up a cluster, validate your actual workload profile. Run benchmarks with realistic message sizes and subscription patterns, not synthetic throughput tests. Engage your team in hands-on labs to build intuition for BookKeeper recovery and namespace policies. If you need guidance architecting or migrating to Pulsar, or want to assess whether it fits your specific compliance and scale requirements, reach out to discuss your streaming infrastructure needs.