Table of Contents
By Khimananda Oli | Last reviewed: August 2026
Debugging distributed systems by SSHing into individual servers is unsustainable and insecure at scale. Centralized logging with the ELK Stack solves this by aggregating logs from every node into a single searchable index, enabling rapid incident response and compliance auditing. This architecture transforms raw text streams into structured observability data, allowing engineering teams to correlate events across microservices without manual correlation. For teams moving beyond basic VPS setups, understanding this pipeline is foundational; if you are still managing single-server deployments, review our guide on securing a fresh Ubuntu VPS before architecting a logging cluster.
How do you configure Filebeat for reliable log shipping?
The foundation of centralized logging with the ELK Stack is lightweight ingestion. Filebeat has largely replaced heavy forwarders because it uses minimal memory and supports backpressure protocols that prevent overwhelming your Logstash instance during traffic spikes. In production environments I manage across AWS and on-premise data centers, misconfigured shippers cause more outages than any other component.
Essential Filebeat Configuration
Avoid wildcard paths in production as they can inadvertently ingest rotated logs or temporary files. Explicitly define inputs and enable multiline parsing for stack traces, which is critical for Java or Python applications where errors span multiple lines.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
service: nginx-web
env: production
fields_under_root: true
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["logstash.internal:5044"]
ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
loadbalance: true
compression_level: 3
processors:
- add_host_metadata: ~
- drop_fields:
fields: ["agent.ephemeral_id", "ecs.version"]- Load balancing: Always set
loadbalance: truewhen pointing to multiple Logstash nodes behind a load balancer to distribute network I/O evenly. - Metadata enrichment: Add host metadata at the shipper level rather than in Logstash to reduce processing overhead on the central pipeline.
- Security: Never ship logs over plaintext TCP in 2026. Use mTLS between Beats and Logstash, especially in multi-tenant cloud environments.
What makes an efficient Logstash pipeline configuration?
Logstash acts as the transformation engine in centralized logging with the ELK Stack. A common mistake is treating it as a simple pass-through; inefficient Grok patterns here will bottleneck your entire observability platform. I have seen pipelines drop 40% of logs during peak hours simply because regex matching was unoptimized.
Pipeline Design Principles
Structure your pipeline to fail gracefully. Use conditional logic to route different log types to specific parsers, and always include a fallback output for unparsed messages so you never lose data silently.
input {
beats {
port => 5044
ssl_enabled => true
ssl_certificate => "/etc/logstash/certs/server.crt"
ssl_key => "/etc/logstash/certs/server.key"
}
}
filter {
if [service] == "nginx-web" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
tag_on_failure => ["_grokparsefailure_nginx"]
}
geoip { source => "clientip" }
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
} else {
mutate { add_tag => ["unparsed_log"] }
}
}
output {
elasticsearch {
hosts => ["https://es-node-1:9200"]
index => "logs-%{[service]}-%{+yyyy.MM.dd}"
user => "${ES_USER}"
password => "${ES_PASS}"
}
}How should Elasticsearch indices be managed for log retention?
Storage costs and query performance are the primary constraints in centralized logging with the ELK Stack. Without lifecycle management, your cluster will eventually run out of disk space or degrade to unusable latency. Index Lifecycle Management (ILM) automates the transition of logs from hot NVMe storage to warm HDD tiers and eventual deletion.
Implementing ILM Policies
Define policies that align with your compliance requirements. For SOC 2 or ISO 27001 audits, you may need to retain security logs for 12 months while keeping application debug logs for only 30 days. Separate these concerns at the index template level.
PUT _ilm/policy/logs-retention-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": { "max_size": "50gb", "max_age": "1d" },
"set_priority": { "priority": 100 }
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 },
"set_priority": { "priority": 50 }
}
},
"delete": {
"min_age": "90d",
"actions": { "delete": {} }
}
}
}
}| Phase | Storage Tier | Typical Retention | Optimization Action |
|---|---|---|---|
| Hot | NVMe SSD | 0–7 Days | Rollover at 50GB, high replica count |
| Warm | HDD / S3 Glacier | 7–90 Days | Force merge, shrink shards, read-only |
| Cold | Object Storage | 90–365 Days | Frozen tier, searchable snapshots |
| Delete | N/A | > Compliance Window | Automated purge via ILM |
When should you choose Fluentd vs Logstash for log aggregation?
While this guide focuses on centralized logging with the ELK Stack, engineers often ask about Fluentd as an alternative processor. The choice depends on your ecosystem and resource constraints. Logstash offers richer plugin support for Elastic-specific features, while Fluentd excels in Kubernetes-native environments with lower memory footprints.
If you are running a pure Kubernetes stack with minimal transformation needs, Fluent Bit as a DaemonSet paired with Elasticsearch is viable. However, for hybrid environments involving legacy apps, Windows servers, or complex parsing rules, Logstash remains the superior choice due to its mature codec library and JDBC streaming capabilities. Teams adopting Kubernetes basics often start with Fluent Bit and migrate to Logstash as compliance requirements grow.
How do you secure the ELK Stack against unauthorized access?
Logs contain sensitive data including PII, API keys, and internal IP addresses. Securing centralized logging with the ELK Stack is not optional; it is a prerequisite for passing any security audit. In my experience helping Nepali fintechs achieve ISO 27001 certification, logging infrastructure is frequently the first system auditors inspect for data leakage risks.
- Encrypt everything: Enable TLS 1.3 for all inter-node communication and client connections. Use certificate-based authentication instead of basic auth wherever possible.
- Field-level redaction: Configure Logstash filters to mask credit card numbers, emails, and tokens before indexing. Once sensitive data hits Elasticsearch, it is effectively permanent until reindexed.
- Role-based access control: Create separate Kibana spaces and Elasticsearch roles for dev, ops, and security teams. Developers should see app logs but not IAM audit trails.
- Network segmentation: Place Elasticsearch and Logstash in private subnets. Only expose Kibana via reverse proxy with WAF protection. Refer to our AWS hosting guide for VPC architecture patterns that apply equally to logging clusters.
Conclusion
Building effective centralized logging with the ELK Stack requires disciplined configuration across ingestion, processing, storage, and security layers. Start with proper Filebeat hygiene, optimize your Logstash pipelines to avoid bottlenecks, implement ILM early to control costs, and treat log security as a first-class concern. The investment pays dividends during incidents and audits alike. If your team needs assistance designing a compliant, scalable logging architecture tailored to your infrastructure, reach out to discuss your observability requirements.
