Table of Contents
By Khimananda Oli | Last reviewed: August 2026
When standard logs and metrics fail to explain intermittent latency or silent failures, you must trace system calls with bpftrace to observe kernel-user interactions directly. Unlike legacy strace methods that impose massive overhead, eBPF-based tracing offers safe, programmable visibility into production Linux systems without restarting services. This guide provides battle-tested scripts and safety patterns derived from years of debugging high-traffic infrastructure.
bpftrace -e command. Use built-in variables like comm, pid, and args to filter output safely. Always prefer tracepoints over kprobes for stability, and limit output frequency in production to avoid perf ring buffer overload.How do you safely trace system calls with bpftrace in production?
Production environments demand caution. While it is powerful, an unbounded trace can saturate CPU or fill buffers faster than userspace can consume them. When I audit infrastructure for Ubuntu server security best practices, I treat dynamic tracing as a privileged operation requiring strict scoping.
Verify kernel support and permissions
Before writing a single line of code, confirm your kernel supports BPF Type Format (BTF). Modern distributions like Ubuntu 24.04+ and RHEL 9+ ship this by default. BTF allows bpftrace to understand kernel structures without manual header parsing, making scripts portable across kernel versions.
# Check for BTF support
ls /sys/kernel/btf/vmlinux
# Verify bpftrace version (aim for 0.21+ in 2026)
bpftrace --version
# Ensure CAP_BPF and CAP_PERFMON capabilities
sudo setcap cap_bpf,cap_perfmon+ep /usr/bin/bpftraceApply mandatory safety guardrails
Never run unbounded traces on high-frequency syscalls like read, write, or futex without filters. The perf ring buffer is finite; overflowing it causes lost events and potential backpressure on the kernel. In my experience managing SOC 2 compliant environments, we enforce these rules via policy-as-code:
- PID Filtering: Always scope to a specific process ID or cgroup during initial investigation.
- Rate Limiting: Use
nsecsdeltas or map-based throttling to cap output to <1000 lines/sec. - Timeout Enforcement: Wrap commands in
timeout 30s bpftrace ...to prevent accidental infinite runs. - Tracepoint Preference: Prefer
tracepoint:syscalls:*overkprobe:*. Tracepoints are stable ABI; kprobes can crash if internal struct layouts change between kernel updates.
What are the most useful bpftrace one-liners for debugging syscalls?
You rarely need to write complex programs from scratch. Most production issues map to known patterns. These one-liners have solved more incidents in my career than any dashboard. They work on any modern Linux distro with BTF support.
Identify processes causing file open errors
Silent failures often manifest as ENOENT or EACCES. This script catches every failed openat call, printing the process name, PID, filename, and error code. It’s invaluable when debugging permission issues in containerized apps where file permissions get tangled across namespaces.
sudo bpftrace -e '
tracepoint:syscalls:sys_exit_openat
/args->ret < 0/ {
printf("%-6d %-16s %s (err=%d)\n",
pid, comm, str(args->filename), args->ret);
}'Measure syscall latency distributions
Averages lie. When debugging storage latency, you need percentiles. This script uses a histogram map to show the distribution of fsync durations. If your p99 spikes while p50 stays flat, you have a tail-latency problem that aggregate metrics hide.
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_fsync { @start[pid] = nsecs; }
tracepoint:syscalls:sys_exit_fsync
/@start[pid]/ {
@duration_us = hist((nsecs - @start[pid]) / 1000);
delete(@start[pid]);
}'Track network socket creation by process
Security audits frequently require proving which binaries initiate outbound connections. This traces socket syscalls, filtering for TCP (type=1) and IPv4/IPv6 families. Pair this with network policies validation to detect policy violations before attackers exploit them.
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_socket
/args->family == AF_INET || args->family == AF_INET6/ {
printf("%-6d %-16s socket(family=%d, type=%d)\n",
pid, comm, args->family, args->type);
}'How does bpftrace compare to strace and perf for syscall tracing?
Choosing the wrong tool wastes hours. Each has a distinct niche based on overhead, safety, and data granularity. Understanding these trade-offs prevents production incidents caused by observer effects.
| Feature | bpftrace | strace | perf trace |
|---|---|---|---|
| Overhead | Low (<5% typically) | Very High (100x+ slowdown) | Moderate (10-30%) |
| Safety | Sandboxed verifier | Ptrace stops process | Safe but limited filtering |
| Programmability | Full C-like language + maps | None (raw stream only) | Limited event selection |
| Aggregation | In-kernel histograms/counts | Post-process required | Basic stats only |
| Best For | Production debugging, custom metrics | Dev environment repro | Sampling hot paths |
| Risk Level | Low (with guardrails) | Critical (never in prod) | Medium |
In practice, I use strace only on isolated dev boxes to reproduce bugs. For any production system, especially those handling customer data under compliance frameworks, bpftrace is the default. Its in-kernel aggregation means you extract insights without shipping terabytes of raw events to userspace—a critical distinction when comparing observability signals at scale.
How do you build custom bpftrace scripts for advanced syscall analysis?
One-liners solve known problems. Custom scripts solve yours. When investigating complex issues like database connection pool exhaustion or microservice timeout cascades, you need multi-probe correlation and state tracking.
Correlate connect() and close() for leak detection
Socket leaks manifest as growing file descriptor counts. This script tracks open sockets per process and flags those held longer than 60 seconds—a common sign of missing cleanup in error paths. I’ve used similar logic to debug Node.js services where Node.js installations lacked proper signal handlers.
#!/usr/bin/env bpftrace
tracepoint:syscalls:sys_enter_connect {
@sock_start[pid, args->fd] = nsecs;
}
tracepoint:syscalls:sys_exit_close
/@sock_start[pid, args->fd]/ {
$dur_ms = (nsecs - @sock_start[pid, args->fd]) / 1000000;
if ($dur_ms > 60000) {
printf("LEAK: pid=%d fd=%d held %d ms\n",
pid, args->fd, $dur_ms);
}
delete(@sock_start[pid, args->fd]);
}
interval:s:10 {
print(@sock_start);
}Profile syscall patterns by cgroup
In Kubernetes, PID-based filtering breaks due to container restarts. Cgroup-based tracing survives pod rescheduling. This aggregates syscall counts per cgroup, revealing noisy neighbors or runaway containers without modifying pod specs.
sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter {
@syscalls[cgroup_path(cgroup)] = count();
}
interval:s:5 {
print(@syscalls, top=10);
clear(@syscalls);
}'Debug DNS resolution latency end-to-end
DNS slowness kills SLOs. This correlates connect to port 53 with subsequent recvfrom latency, giving true resolver time excluding application processing. Combine with DNS configuration guides to validate fixes quantitatively.
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_connect
/args->uservaddr->sin_port == 53/ {
@dns_start[pid] = nsecs;
}
tracepoint:syscalls:sys_exit_recvfrom
/@dns_start[pid]/ {
$lat_us = (nsecs - @dns_start[pid]) / 1000;
@dns_latency_us = hist($lat_us);
delete(@dns_start[pid]);
}'Start Tracing System Calls Safely Today
Tracing system calls with bpftrace transforms opaque kernel behavior into actionable data, but respect its power. Begin with filtered one-liners on non-critical hosts, validate output rates, and graduate to custom scripts only after mastering safety primitives. Your production systems will thank you for the discipline. If you need help designing observability strategies that survive audits and traffic spikes alike, reach out to discuss your infrastructure.
