The ELK Stack: Elasticsearch, Logstash, Kibana

Q: What is the ELK Stack used for in 2026?

The ELK Stack centralizes log aggregation, search, and visualization. Elasticsearch indexes data, Logstash parses pipelines, and Kibana provides dashboards for monitoring infrastructure, application performance, and security events across distributed systems.

Q: Is the ELK Stack still free to use?

Yes, the Basic license remains free and open source under SSPL/Elastic License 2.0. It includes core Elasticsearch, Logstash, and Kibana features but excludes advanced security, alerting, and machine learning capabilities found in paid tiers.

Q: How much RAM does Elasticsearch need for production?

Allocate at least 31GB heap maximum per node, leaving remaining system RAM for Lucene filesystem cache. Production clusters typically require 64GB+ total RAM per node to maintain performance and avoid garbage collection pauses during heavy indexing.

Q: What replaced Logstash in modern ELK deployments?

Filebeat and Elastic Agent now handle lightweight log shipping and parsing. Logstash remains useful for complex transformations, enrichment, or routing multiple inputs to diverse outputs, but most edge collection uses Beats to reduce resource overhead significantly.

Q: How do I secure Elasticsearch without X-Pack?

Use reverse proxies like Nginx with basic auth, enable TLS encryption between nodes, restrict network binding to localhost or private interfaces, and implement firewall rules. Consider OpenSearch Security plugin as a free alternative for role-based access control.

Q: Why is my Elasticsearch cluster status yellow?

Yellow indicates unassigned replica shards, usually from insufficient data nodes. Add more nodes matching your replica count or reduce replica settings for single-node development environments. Check shard allocation awareness and disk watermark thresholds preventing assignment.

Q: Can Kibana connect to multiple Elasticsearch clusters?

No, each Kibana instance connects to exactly one Elasticsearch cluster. Deploy separate Kibana instances for each cluster or use cross-cluster search in Elasticsearch to query remote indices through a single coordinating cluster and Kibana interface.

Q: What are common Logstash pipeline bottlenecks?

Grok pattern failures, insufficient worker threads, slow external lookups, and backpressure from downstream outputs cause bottlenecks. Monitor pipeline throughput via monitoring APIs, increase workers based on CPU cores, use persistent queues, and profile filter performance regularly.

Q: How do I rotate Elasticsearch indices automatically?

Use Index Lifecycle Management policies to automate rollover, shrink, force merge, and deletion based on age or size. Define hot-warm-cold phases matching your retention requirements and attach ILM policies to index templates during creation.

Q: Does ELK Stack work on Kubernetes?

Yes, Elastic Cloud on Kubernetes operator manages stateful sets, configmaps, and services declaratively. ECK handles upgrades, scaling, TLS certificates, and snapshot repositories natively while integrating with Prometheus metrics and service mesh observability patterns.

August 11, 2026 Khimananda Oli 4 min read Virtualization

By Khimananda Oli | Last reviewed: August 2026

The ELK Stack: Elasticsearch, Logstash, Kibana remains the industry-standard open-source foundation for centralized logging, enabling teams to aggregate, search, and visualize logs across distributed systems in real time. While managed services exist, self-hosting this stack gives you full control over data residency, retention policies, and cost—critical factors for organizations operating under strict compliance frameworks or budget constraints. This guide covers the production-grade deployment patterns I use daily, moving beyond basic tutorials to address the architectural decisions that actually determine success.

Quick answer: The ELK Stack: Elasticsearch, Logstash, Kibana is an integrated observability platform where Elasticsearch stores and indexes logs, Logstash transforms and routes data through pipelines, and Kibana provides visualization. In 2026, most production deployments add Beats agents at the edge and often replace Logstash with lighter shippers for simple ingestion tasks.

What Is the ELK Stack: Elasticsearch, Logstash, Kibana Architecture?

Understanding the distinct role of each component prevents the most common failure mode: treating the stack as a monolith. Elasticsearch is a distributed search and analytics engine built on Apache Lucene; it is not a database in the traditional sense and should never be used as a primary data store. Logstash is a server-side processing pipeline with input, filter, and output plugins that can parse, enrich, and route events. Kibana is strictly a visualization layer that queries Elasticsearch via its REST API—it holds no data itself.

Core architecture of the ELK Stack: Elasticsearch, Logstash, Kibana with Beats as the recommended edge shipper

In practice, the modern stack almost always includes Beats (Filebeat, Metricbeat, Auditbeat) as lightweight shippers installed directly on source hosts. Sending logs directly from application servers to Logstash creates tight coupling and single points of failure. Instead, Beats handle local file tailing, backpressure, and acknowledgment protocols, forwarding structured batches to Logstash or directly to Elasticsearch for simpler workloads. For teams exploring broader observability strategies, understanding how observability differs from traditional monitoring helps justify the investment in this architecture.

How Do You Configure Secure Communication Between ELK Components?

Security is not optional. I have audited too many environments where Elasticsearch was exposed on 0.0.0.0:9200 without authentication—a critical vulnerability that has led to real-world data breaches. Every production deployment of the ELK Stack: Elasticsearch, Logstash, Kibana must enforce TLS encryption between nodes and enable X-Pack security (now included in the basic license).

Enable Built-in Security Features

Add these settings to elasticsearch.yml on every node before first start:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path: certs/http-certs.p12
discovery.type: multi-node

Generate certificates using the bundled tool:

bin/elasticsearch-certutil ca --out elastic-stack-ca.p12 --pass ""
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 \
  --ca-pass "" --out elastic-certificates.p12 --pass "" \
  --name es-node-01 --dns es-node-01.internal --ip 10.0.1.10

Distribute the PKCS#12 files securely and set passwords via the keystore command rather than plaintext config. For Kibana, configure kibana.yml with matching truststore paths and service account tokens instead of legacy user credentials. Never disable SSL verification in production; if certificate validation fails, fix the certificate chain—not the check.

Network-Level Controls

Bind Elasticsearch HTTP to internal interfaces only (network.host: _site_)
Use firewall rules to restrict port 9200/9300 access to known Logstash/Kibana IPs
Place Kibana behind a reverse proxy (Nginx/Caddy) for additional rate limiting and WAF protection
Implement IP filtering in Elasticsearch for multi-tenant clusters

For organizations handling sensitive financial or health data in Nepal or globally, align these controls with your compliance framework. My guide on data protection basics for fintech maps these technical controls to regulatory requirements.

When Should You Use Beats vs Logstash for Log Ingestion?

This decision impacts both operational complexity and resource consumption. A common mistake is routing everything through Logstash out of habit, when Beats alone suffices for 70% of use cases.

Criteria	Beats Direct to ES	Beats → Logstash → ES
Parsing Complexity	Simple JSON, structured logs	Grok, mutate, geoip, external enrichment
Resource Overhead	<50 MB RAM per host	1–4 GB JVM heap minimum
Fan-out / Routing	Single destination only	Conditional outputs (ES + S3 + Kafka)
Backpressure Handling	Built-in disk queue
Maintenance Burden	Config file per host type	Centralized pipeline management
Best For	App logs, nginx access, audit trails	Legacy formats, PII redaction, multi-sink

In my experience, start with Filebeat shipping directly to Elasticsearch using ingest pipelines for basic parsing. Only introduce Logstash when you need conditional logic, external lookups (e.g., enriching IPs against a CMDB), or simultaneous output to multiple destinations. This keeps your infrastructure leaner and easier to troubleshoot during incidents.

Decision framework for selecting ingestion path in the ELK Stack: Elasticsearch, Logstash, Kibana deployments

How Do You Optimize Elasticsearch Performance for High-Volume Logging?

Elasticsearch performance degrades predictably when misconfigured. These are the levers that matter most in production logging workloads:

Index Lifecycle Management (ILM): Define hot-warm-cold-delete phases explicitly. Hot indices accept writes on fast NVMe; warm tier handles searches on cheaper storage; cold tier uses frozen snapshots. Without ILM, you will either run out of disk or overpay for premium storage on stale data.
Shard Sizing: Target 30–50 GB per primary shard for logging. Too many small shards exhaust cluster state memory; too few large shards prevent parallelism. Calculate based on daily volume × retention days ÷ target shard size.
Refresh Interval: Increase from default 1s to 30s or 60s for pure logging indices. Near-real-time search is rarely needed for logs, and this change can double indexing throughput.
Field Mapping Discipline: Disable _source storage for high-cardinality fields you never retrieve. Use keyword over text for structured fields like hostnames and status codes. Set ignore_above: 256 to prevent mapping explosions from malformed log lines.
JVM Heap: Never exceed 31 GB or 50% of system RAM (whichever is lower). Leave adequate memory for Lucene filesystem cache—this is what makes searches fast on large datasets.

Monitor these metrics continuously: indexing rate vs. rejection count, search latency percentiles, GC pause duration, and disk watermark breaches. When integrating with AI-driven analysis tools, ensure your index templates preserve the field structures those tools expect, as discussed in AI-powered log analysis.

What Are Common Production Pitfalls and How Do You Avoid Them?

After deploying and rescuing dozens of ELK environments, these failures recur consistently:

Unbounded Field Cardinality: User IDs, request UUIDs, or session tokens mapped as keywords create millions of unique terms, exhausting heap and crashing nodes. Always profile new fields with the _field_caps API before promoting to production mappings.

Missing Backpressure: Logstash pipelines without persistent queues lose events during restarts or downstream slowdowns. Enable queue.type: persisted and configure dead letter queues for unprocessable events. Similarly, ensure Filebeat has disk spool enabled.

Kibana Query Abuse: Unrestricted dashboard queries scanning months of data freeze browsers and overload coordinating nodes. Enforce time-range defaults, use rollup indices for historical aggregations, and implement query timeouts at the proxy layer.

Version Mismatch: All three components must share the same major version. Mixing Elasticsearch 8.x with Kibana 7.x causes silent failures and unsupported API calls. Pin versions in your IaC and test upgrades in staging first.

Health comparison matrix for production ELK Stack: Elasticsearch, Logstash, Kibana operations

Deploying the ELK Stack: Elasticsearch, Logstash, Kibana With Confidence

Successful deployment of the ELK Stack: Elasticsearch, Logstash, Kibana hinges on treating it as a distributed system requiring deliberate capacity planning, security hardening, and operational discipline—not a plug-and-play appliance. Start with Beats direct ingestion, enforce TLS and RBAC from day one, implement ILM before your first terabyte, and monitor the health indicators that actually predict failure. When your logging volume grows or compliance demands tighten, the foundation you build now determines whether scaling is routine or catastrophic.

If you need hands-on support designing, securing, or optimizing your logging infrastructure for production workloads, reach out to discuss your specific environment. I help teams deploy observability stacks that survive audits, traffic spikes, and 3 AM incident calls.

Frequently Asked Questions