Event-Driven Microservices with Kafka

By Khimananda Oli | Last reviewed: August 2026

Decoupling synchronous HTTP calls is the primary challenge when scaling distributed systems, and implementing event-driven microservices with Kafka solves this by replacing tight coupling with asynchronous message passing. Instead of service A waiting for service B to respond, services publish state changes to topics that interested consumers process independently. This guide provides the exact configuration patterns, producer logic, and consumer group strategies I use in production environments to ensure reliability and observability.

How do you configure event-driven microservices with Kafka for production reliability?

Default Kafka configurations are tuned for throughput, not safety. In production, especially when handling financial transactions or user data subject to compliance frameworks like SOC 2 or ISO 27001, you must explicitly configure durability and ordering guarantees. A common mistake is leaving acks=1 (the default), which acknowledges a write once only the leader has persisted it; if the leader crashes before replication, that event is lost forever.

Essential Producer Configuration

Your producer config determines whether your event-driven microservices with Kafka can survive broker failures without data loss. Set these properties in your application.yml or environment variables:

spring:
  kafka:
    producer:
      acks: all
      retries: 2147483647
      max-in-flight-requests-per-connection: 5
      enable-idempotence: true
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        delivery.timeout.ms: 120000
        request.timeout.ms: 30000
        transactional.id: order-service-tx-${HOSTNAME}

• acks=all: Requires every in-sync replica (ISR) to acknowledge the write. Combined with min.insync.replicas=2 on the topic, this guarantees no acknowledged event is lost even if one broker dies.
• enable-idempotence=true: Prevents duplicate writes during network retries by assigning each producer a unique ID and sequence numbers per partition.
• transactional.id: Enables exactly-once semantics across multiple partitions. Each producer instance needs a stable, unique ID tied to its hostname or pod name.

Topic Design for Ordering and Scale

Partition count dictates your maximum parallelism. If you have 12 consumer instances but only 6 partitions, half your consumers sit idle. I typically start with partitions equal to expected peak consumer count × 1.5 to allow headroom. Use a meaningful key (like orderId) to guarantee all events for the same entity land in the same partition, preserving causal order.

bin/kafka-topics.sh --create \
  --topic orders.created \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000 \
  --bootstrap-server kafka-broker-1:9092

For teams managing infrastructure declaratively, defining topics via Infrastructure as Code with Terraform prevents configuration drift between staging and production environments.

How do consumers handle failures and maintain offset integrity?

The most dangerous failure mode in event-driven microservices with Kafka is processing an event successfully but failing to commit the offset, causing reprocessing on restart. Conversely, committing before processing completes risks skipping events entirely. The solution is manual acknowledgment combined with idempotent consumer logic.

Manual Acknowledgment Pattern

Disable auto-commit and acknowledge only after your database transaction succeeds. This ensures at-least-once delivery:

@KafkaListener(
    topics = "orders.created",
    groupId = "inventory-service",
    ackMode = "MANUAL_IMMEDIATE"
)
public void handleOrderCreated(
    @Payload OrderCreatedEvent event,
    @Header(KafkaHeaders.ACKNOWLEDGMENT) Acknowledgment ack
) {
    try {
        inventoryService.reserveStock(event);
        ack.acknowledge(); // Commit ONLY after successful processing
    } catch (OptimisticLockException e) {
        log.warn("Concurrency conflict for order {}, will retry", event.orderId());
        throw e; // Triggers retry; offset NOT committed
    } catch (Exception e) {
        log.error("Permanent failure for order {}", event.orderId(), e);
        deadLetterService.send(event, e);
        ack.acknowledge(); // Commit to avoid infinite redelivery
    }
}

Idempotency Is Non-Negotiable

Even with manual acks, duplicates occur during rebalances or transient failures. Every consumer must be idempotent. Store processed event IDs in a deduplication table within the same database transaction as your business logic. Check this table before processing. For teams already running containerized workloads, understanding Docker fundamentals helps isolate consumer instances and manage dependencies cleanly during local development and testing.

How does Kafka compare to RabbitMQ and AWS SQS for microservices?

Choosing the right message broker depends on your specific requirements for retention, throughput, and operational complexity. While all three support asynchronous communication, their architectural differences matter significantly at scale.

Kafka wins when you need event history, replayability, or stream processing. RabbitMQ excels at complex routing and low-latency task queues where messages are ephemeral. SQS removes operational overhead entirely but sacrifices replay and ordering flexibility. For Nepal-based startups evaluating cloud costs, comparing AWS vs Azure vs Google Cloud helps determine whether managed Kafka (MSK/Confluent) or self-hosted on EC2 fits your budget and compliance needs.

How do you monitor and observe Kafka pipelines in production?

You cannot fix what you cannot see. Monitoring event-driven microservices with Kafka requires tracking three distinct layers: broker health, producer performance, and consumer lag. Consumer lag is your single most important metric — it measures how far behind consumers are relative to the latest produced offset. Growing lag means your consumers cannot keep up, indicating either insufficient parallelism or slow processing logic.

Critical Metrics to Alert On

1. kafka_consumer_group_lag: Alert when lag exceeds your SLA threshold (e.g., 10,000 messages or 5 minutes). Sustained lag indicates capacity issues.
2. kafka_producer_record_error_rate: Any sustained error rate above 0.1% warrants investigation. Common causes include broker unavailability, serialization failures, or topic misconfiguration.
3. kafka_server_UnderReplicatedPartitions: Should always be zero. Non-zero values mean replicas are falling behind, risking data loss if the leader fails.
4. kafka_controller_KafkaController_OfflinePartitionsCount: Partitions with no active leader. Immediate P1 incident.

Export these metrics via JMX or Kafka Exporter into Prometheus. Pair with Grafana dashboards that overlay producer rates against consumer rates to visualize backpressure. For comprehensive observability setup, follow the Prometheus and Grafana complete setup guide to integrate Kafka metrics alongside application and infrastructure telemetry.

What security controls are mandatory for Kafka in regulated environments?

If your event-driven microservices with Kafka handle personal data, payment information, or anything under SOC 2 / ISO 27001 scope, plaintext traffic and open brokers are unacceptable. Implement defense-in-depth across transport, authentication, and authorization.

• TLS everywhere: Encrypt inter-broker communication (security.inter.broker.protocol=SSL) and client connections (listeners=SSL://0.0.0.0:9093). Use short-lived certificates rotated via cert-manager or Vault PKI.
• mTLS or SASL/SCRAM-SHA-256: Authenticate every client. Avoid plaintext SASL mechanisms. Bind credentials to service identities, not shared secrets.
• ACLs per topic: Grant producers WRITE access only to their owned topics. Grant consumers READ access only to topics they legitimately need. Deny by default.
• Network segmentation: Place Kafka brokers in private subnets. Expose only through controlled ingress points. Never expose broker ports to the public internet.

Store Kafka credentials in HashiCorp Vault or AWS Secrets Manager — never in code, config files, or environment variables baked into container images. Rotate credentials automatically and audit access logs for anomalous patterns. These controls form the baseline for passing external audits without last-minute scrambles.

Next Steps for Your Kafka Implementation

Building reliable event-driven microservices with Kafka requires deliberate configuration choices around durability, ordering, idempotency, and observability — none of which come free from defaults. Start with the producer and consumer configurations outlined here, instrument lag monitoring before your first production deploy, and enforce TLS plus ACLs from day one. If you are designing a new event-driven architecture or hardening an existing Kafka deployment for compliance and scale, reach out to discuss your specific requirements.