
Table of Contents
By Khimananda Oli | Last reviewed: August 2026
Static scanning catches vulnerabilities before deployment, but it cannot stop an attacker who exploits a zero-day or steals credentials post-deploy. Falco: Runtime Security for Kubernetes fills this gap by monitoring system calls in real time to detect active breaches, unauthorized shell access, and data exfiltration attempts. As teams adopt DevSecOps practices, integrating runtime detection becomes the critical last line of defense when preventive controls fail.
How does Falco: Runtime Security for Kubernetes actually work?
Falco operates as a userspace daemon that consumes a stream of system call events from the kernel. Unlike older tools that relied on kernel modules (which risked panics during upgrades), modern Falco defaults to eBPF probes. This approach attaches safely to syscall entry/exit points, capturing arguments, return values, and process context with minimal overhead. The engine evaluates each event against a YAML rule set; matches trigger alerts via stdout, syslog, or webhooks.
The key distinction is visibility: network policies see packets, and admission controllers see manifests, but only runtime security sees behavior. A pod might pass all admission checks yet still execute /bin/sh via an RCE exploit minutes later. Falco catches this because the execve syscall is observable regardless of how the process was spawned. In my experience auditing SOC 2 environments, this behavioral layer is often what separates "compliant on paper" from "actually secure."
How do you install Falco on a production Kubernetes cluster?
Use the official Helm chart. It handles DaemonSet scheduling, eBPF probe compilation, and config mounting. Avoid manual binary installs in production; they complicate upgrades and node scaling.
Add the repo and generate a minimal values file
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
cat > falco-values.yaml <<'EOF'
driver:
kind: ebpf
ebpf:
hostNetwork: false
falco:
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/custom_rules.yaml
json_output: true
http_output:
enabled: true
url: "http://falcosidekick:2801/"
tolerations:
- operator: Exists
EOF Deploy with namespace isolation
kubectl create namespace falco-system
helm upgrade --install falco falcosecurity/falco \
-n falco-system \
-f falco-values.yaml \
--version 4.12.0 \
--wait --timeout 5m Always pin the chart version. Falco 4.x introduced breaking changes to the driver interface; unpinned installs can silently drift during CI runs. Verify pods are running with kubectl get pods -n falco-system -l app.kubernetes.io/name=falco and check logs for "eBPF probe loaded" confirmation. If nodes lack kernel headers, enable the driver.loader.initContainer option to compile probes at startup.
Which Falco rules matter most for detecting real attacks?
The default rule set contains over 300 rules. Most are informational noise in greenfield clusters. Focus first on high-signal detections that map to MITRE ATT&CK tactics relevant to containers.
- Terminal shell in container: Detects interactive shells spawned in non-debug pods. Critical for catching RCE post-exploitation.
- Read sensitive file untrusted: Flags access to
/etc/shadow, SSH keys, or cloud metadata endpoints (169.254.169.254) by unexpected processes. - Outbound connection to C2 ports: Alerts on connections to known malicious ports or unusual DNS queries from application containers.
- Write below monitored directories: Catches binaries dropped into
/bin,/usr/local/bin, or writable PATH entries — common persistence technique. - K8s secret access from pod: Monitors API server calls for secret retrieval outside expected service accounts.
Create custom rules in a separate file to survive upgrades. Use append: true to extend lists without overriding upstream definitions:
- rule: Terminal Shell in Production Namespace
desc: Detect shell spawn in prod namespaces excluding debug pods
condition: >
spawned_process and container and
proc.name in (bash, sh, zsh, dash) and
k8s.ns.name startswith "prod-" and
not k8s.pod.label["app"] = "debug-toolbox"
output: "Shell spawned in prod container (user=%user.name container=%container.name cmd=%proc.cmdline)"
priority: WARNING
tags: [mitre_execution, soc2] Test rules locally before deploying. Use falco -U -r custom_rules.yaml against recorded syscall captures to validate logic without generating production noise. For deeper context on securing cluster access patterns that trigger these rules, review Kubernetes RBAC best practices.
How do you reduce Falco alert fatigue without missing breaches?
Untuned Falco generates hundreds of alerts per hour in active clusters. Alert fatigue causes teams to disable the tool entirely. Apply these filters systematically:
- Baseline normal behavior first. Run Falco in dry-run mode for 7 days. Export events to object storage. Identify top 20 noisy rules and determine if they represent expected platform behavior (e.g., node exporters reading
/proc). - Use exceptions, not deletions. Never delete upstream rules. Add
exceptionsblocks to whitelist known-good processes by executable path, container image, or namespace label. This preserves detection coverage for unknown variants. - Tag rules by compliance framework. Group rules under
soc2,pci, ormitre_*tags. Enable/disable tag sets per environment. Staging can run verbose rules; production runs only tagged high-priority sets. - Implement rate limiting at the sink. Configure Falcosidekick deduplication windows. Identical alerts within 60 seconds collapse to one notification. Prevents Slack/PagerDuty storms during incidents.
- Correlate with identity context. Enrich alerts with pod labels, service account names, and node metadata. An alert from
payment-apipod withsa=payment-processorcarries different urgency than one from an unnamed test pod.
A common mistake is silencing rules globally because they're noisy in one namespace. Use scoped exceptions instead. For example, allow curl in the monitoring namespace but keep the rule active elsewhere. This maintains security posture while respecting operational reality.
Falco vs other Kubernetes security tools: When is runtime detection necessary?
Runtime security overlaps with admission control and network policy. Understanding the boundaries prevents redundant spending and gaps.
| Capability | Falco (Runtime) | Admission Controllers (OPA/Kyverno) | Network Policy (Cilium/Calico) |
|---|---|---|---|
| Detects post-deploy exploitation | Yes (syscall-level) | No (pre-deploy only) | Partial (network-layer only) |
| Blocks malicious activity | No (detect-only)* | Yes (reject manifests) | Yes (drop packets) |
| Sees encrypted traffic content | No | N/A | No (L3/L4 only) |
| Detects insider misuse of valid creds | Yes | No | No |
| Performance overhead | Low (eBPF, ~2-5%) | Minimal | Low |
| Compliance evidence generation | Runtime audit logs | Policy-as-code proofs | Network flow logs |
*Falco can integrate with response engines like KubeArmor or Tekton triggers for automated containment, but core Falco is detection-focused.
In practice, you need all three. Admission prevents misconfigurations. Network policy limits blast radius. Falco detects when both fail. For teams pursuing SOC 2 or ISO 27001, runtime monitoring satisfies specific control requirements around intrusion detection that static controls alone cannot fulfill. Pair Falco alerts with centralized logging via structured logging pipelines to ensure audit-ready evidence retention.
Integrating Falco Alerts Into Your Observability Stack
Falco outputs JSON. Route it through Falcosidekick to normalize fields and fan out to multiple backends. Direct stdout logging loses structure and makes querying painful.
# falcosidekick-values.yaml
config:
loki:
host: "http://loki.monitoring:3100"
tenant: "falco-runtime"
slack:
webhook: "${SLACK_WEBHOOK_URL}"
channel: "#security-alerts"
minimumpriority: "warning"
prometheus:
enabled: true
port: 2801 Build Grafana dashboards that correlate Falco events with resource metrics. A spike in Terminal Shell alerts coinciding with CPU anomalies on a specific node suggests active compromise, not false positives. Set SLOs around mean-time-to-detect (MTTD); aim for under 60 seconds for critical rules. Document response playbooks linking each high-priority rule to concrete investigation steps. Teams that treat Falco as a fire-and-forget install accumulate debt; those that integrate it into incident workflows gain measurable security ROI.
Next Steps for Production-Grade Runtime Security
Deploying Falco: Runtime Security for Kubernetes is a starting point, not a destination. Begin with the five high-signal rules outlined above, establish a baseline tuning cycle, and integrate alerts into your existing observability stack before expanding coverage. Measure detection latency and false positive rates weekly for the first month. If your team lacks bandwidth to maintain rule hygiene, consider managed detection services or commercial Falco distributions with pre-tuned enterprise rule sets. For architecture reviews or help designing a compliant runtime security program, reach out directly.