Hubble: Network Observability with eBPF

By Khimananda Oli | Last reviewed: August 2026

Debugging silent failures in Kubernetes often feels like guessing because standard metrics lack packet-level context. Hubble: Network Observability with eBPF solves this by attaching directly to the kernel, providing deep visibility into service-to-service communication without application code changes or sidecar proxies. If you are already running Cilium for CNI, Hubble is likely available but underutilized; enabling it transforms opaque cluster networking into a queryable, auditable data stream essential for both troubleshooting and compliance.

How does Hubble: Network Observability with eBPF differ from traditional monitoring?

Traditional Kubernetes monitoring relies on two primary sources: application-instrumented metrics (OpenTelemetry, Prometheus client libraries) and infrastructure metrics from cAdvisor or node-exporter. Both have blind spots. Application metrics only exist if developers correctly instrumented the code; infrastructure metrics show resource usage but not communication semantics. You might see high CPU on a pod, but you cannot tell if it is processing legitimate requests or being hammered by a misconfigured upstream service retrying indefinitely.

Hubble operates at a fundamentally different layer. By leveraging eBPF programs attached to TC (Traffic Control) hooks or XDP (eXpress Data Path), it observes every packet entering or leaving a container's network namespace. This happens entirely within the kernel, independent of the application runtime. The result is ground-truth visibility that works for legacy applications, third-party containers, and system services alike.

Key architectural distinctions

• No sidecar proxy required: Unlike Istio or Linkerd, Hubble does not intercept traffic through an Envoy sidecar. This eliminates the "double-hop" latency penalty and reduces memory overhead per pod significantly. For teams managing lightweight service meshes, Hubble provides comparable L7 visibility without the operational tax.
• Identity-aware flows: Raw packet captures show IP addresses and ports. Hubble enriches every flow event with Kubernetes metadata: source/destination pod names, namespaces, labels, and even DNS names. This enrichment happens at the agent level before data leaves the node, making queries human-readable immediately.
• Policy verdict correlation: When using Cilium NetworkPolicies, Hubble records whether a packet was allowed or denied by a specific policy rule. This closes the feedback loop between security intent and actual enforcement, which is critical when validating network policy configurations.

How do you install and configure Hubble in a production cluster?

If you are deploying Cilium as your CNI, enable Hubble during installation rather than retrofitting it later. The Helm chart handles component deployment, certificate generation for mTLS between agents and relay, and service exposure.

helm upgrade --install cilium cilium/cilium \
  --namespace kube-system \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,http}" \
  --set hubble.metrics.serviceMonitor.enabled=true

The metrics.enabled flag is where most operators make mistakes. By default, Hubble collects flow events for the UI and CLI but does not expose Prometheus metrics unless explicitly configured. Always enable the specific metric families you need; collecting all metrics on high-throughput clusters can generate significant cardinality.

Validating the deployment

After installation, verify that the Hubble Relay is receiving flows from all nodes. A common failure mode in multi-zone or hybrid clusters is firewall rules blocking the peer-to-peer gRPC port (4244/TCP) between Cilium agents.

# Check relay connectivity status
kubectl exec -n kube-system deploy/hubble-relay -- hubble status

# Expected output should show connected nodes matching your cluster size
# Current Status: Connected to 12/12 nodes

For local development or testing, use the Hubble CLI directly against the relay port-forward. Avoid exposing the Hubble UI publicly without authentication; it contains sensitive topology and traffic data that violates least-privilege principles.

What are the most effective Hubble CLI commands for debugging production incidents?

The Hubble CLI is purpose-built for incident response. Unlike generic packet capture tools, its filters understand Kubernetes primitives. Memorize these patterns; they replace hours of log grepping during outages.

1. Identify dropped traffic by namespace:

   hubble observe --namespace payments --verdict DROPPED --output json | jq '.flow.reason'

   This immediately surfaces whether drops are caused by network policies, missing DNS resolution, or kernel-level errors. In SOC 2 audit scenarios, this command provides evidence that unauthorized cross-namespace traffic is being blocked as designed.
2. Trace HTTP errors between specific services:

   hubble observe --from-pod checkout-service --to-service inventory-api \
     --type http:http-status-code-gte=400 --follow

   Filtering by HTTP status code at the kernel level means you see errors even if the application fails to log them. This is invaluable when debugging intermittent 502/503 errors that disappear from application logs due to buffer flushing issues.
3. Map external egress dependencies:

   hubble observe --namespace billing --to-fqdn '*.stripe.com' --type dns

   Before migrating workloads or tightening egress policies, use this to discover actual external dependencies. Teams frequently find forgotten webhook endpoints or legacy API calls that are not documented in architecture diagrams.
4. Correlate DNS failures with connection timeouts:

   hubble observe --type dns:rcode!=0 --last 50

   DNS resolution failures are the silent killer of microservices. This filter catches NXDOMAIN and SERVFAIL responses that cause downstream timeout cascades. Pair this with structured logging to correlate DNS failures with application error spikes.

How does Hubble compare to service mesh observability and standalone eBPF tools?

Choosing the right observability tool depends on your existing stack and operational constraints. Hubble occupies a specific niche: deep network visibility tightly integrated with Cilium's dataplane. Understanding trade-offs prevents over-engineering.

In practice, if you are already committed to Cilium, Hubble provides 80% of service mesh observability value at 10% of the cost. Reserve full service mesh adoption for cases requiring advanced traffic splitting, circuit breaking, or multi-cluster failover. For teams focused purely on runtime security without Cilium, Tetragon offers similar eBPF capabilities with stronger process-level enforcement, though it lacks Hubble's seamless Kubernetes identity enrichment.

Integrating Hubble Metrics with Your Existing Observability Stack

Hubble's true power emerges when combined with your broader monitoring ecosystem. Flow events alone are valuable for ad-hoc debugging, but persistent metrics enable trend analysis, alerting, and SLO tracking. Configure the ServiceMonitor during Helm installation to automatically integrate with Prometheus and Grafana stacks.

Focus on three high-value metric families first. hubble_flows_processed_total broken down by verdict and reason gives you a baseline for normal traffic patterns; deviations signal policy misconfigurations or attacks. hubble_dns_queries_total with response code labels catches DNS degradation before users report slowness. hubble_http_requests_total with status code and method labels provides L7 golden signals without application instrumentation, filling gaps when teams haven't yet adopted OpenTelemetry.

Avoid the trap of enabling every available metric. High-cardinality labels like source/destination pod names create storage explosions. Use recording rules to pre-aggregate at namespace or service level for dashboards, reserving raw flow queries for interactive debugging sessions. This discipline keeps Prometheus queryable and your cloud bill predictable.

Next Steps for Production-Grade Network Observability

Hubble: Network Observability with eBPF represents a fundamental shift in how we understand Kubernetes networking. Start by enabling it in a non-production environment and practicing the CLI workflows described above until they become muscle memory. Then, gradually roll out metric collection aligned with your team's actual debugging pain points rather than theoretical completeness.

Remember that observability tools only deliver value when integrated into incident response runbooks and postmortem processes. Document your Hubble query patterns alongside your golden signals definitions so new engineers inherit institutional knowledge. If your team needs guidance implementing eBPF-based observability or integrating Hubble with existing compliance frameworks, reach out to discuss your specific architecture.