The Four Golden Signals of Monitoring

By Khimananda Oli | Last reviewed: August 2026

Alert fatigue is the silent killer of engineering velocity. When your team ignores PagerDuty notifications because 90% are false positives, you have a signal-to-noise problem, not a tooling problem. Implementing the Four Golden Signals of Monitoring—latency, traffic, errors, and saturation—provides a standardized framework to distinguish genuine user pain from benign infrastructure fluctuations. This approach, popularized by Google’s Site Reliability Engineering practices, remains the most effective baseline for actionable observability in 2026.

What are the Four Golden Signals of Monitoring and why do they matter?

The Four Golden Signals of Monitoring are a minimal, sufficient set of metrics defined in the Google SRE Handbook to assess service health. Unlike generic dashboarding that tracks hundreds of vanity metrics, these four signals focus exclusively on user experience and system limits. In my experience helping teams across Nepal and globally achieve SOC 2 compliance, auditors and engineers alike prefer this model because it maps directly to Service Level Objectives (SLOs).

A common mistake I see in growing startups is monitoring infrastructure in isolation. You might have CPU at 40% and memory at 50%, yet users are experiencing timeouts. Without tracking latency and errors alongside saturation, you miss the actual problem. These signals force you to monitor the service, not just the server. For teams adopting observability versus monitoring strategies, these signals serve as the quantitative foundation upon which qualitative tracing and logging are built.

Defining the signals precisely

• Latency: The time it takes to service a request. Crucially, you must distinguish between successful and failed requests. A fast error is still an error; averaging it with slow successes masks both problems.
• Traffic: A measure of demand on your system. For web services, this is HTTP requests per second. For databases, it might be queries or transactions. Traffic contextualizes the other three signals.
• Errors: The rate of requests that fail. This includes explicit failures (HTTP 5xx) and implicit failures (HTTP 200 with wrong content or excessive latency).
• Saturation: How full your resources are. This includes CPU, memory, disk I/O, and connection pools. Saturation is a leading indicator; once you hit 100%, latency and errors follow immediately.

How do you measure latency and traffic correctly in production?

Measuring latency seems trivial until you encounter p99 outliers. Average latency is useless for SRE because it hides tail latency issues that affect real users. You should always track latency using percentiles (p50, p95, p99) or histograms. In Prometheus, use the histogram_quantile function rather than simple averages.

# Calculate p99 latency for successful requests only
histogram_quantile(0.99, 
  sum(rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le)
)

# Calculate traffic (requests per second)
sum(rate(http_requests_total[5m])) by (service)

Traffic serves as the denominator for your error rate and the context for your latency. If latency spikes but traffic has dropped by 80%, you likely have a scaling-down issue or a background job consuming resources, not a user-facing crisis. Conversely, if traffic doubles and latency increases linearly, your system lacks auto-scaling headroom. Understanding this relationship is critical when configuring Kubernetes autoscaling policies that respond to actual demand rather than arbitrary CPU thresholds.

Distinguishing success from failure in latency

Never aggregate success and error latency into a single metric. Failed requests often return instantly (e.g., a quick 400 Bad Request), which artificially lowers your average latency. Always split your histogram buckets by status code class. In Grafana, visualize success latency and error latency as separate panels or distinct series colors. This separation ensures that a spike in fast failures doesn't mask a degradation in successful response times.

How do you detect implicit errors and track saturation effectively?

Explicit errors are easy: count the 5xx responses. Implicit errors are where experienced engineers earn their keep. An API returning HTTP 200 with an empty JSON array when data was expected is an error. A page loading in 8 seconds when the SLO is 2 seconds is an error. To catch these, you need application-level instrumentation, not just reverse proxy logs.

I recommend implementing semantic validation in your middleware or sidecar. For example, if a checkout endpoint returns 200 but the cart total is zero, increment an implicit_errors_total counter. Combine this with your explicit error counter for the true error rate:

# True error rate including implicit failures
(
  sum(rate(http_requests_total{status=~"5.."}[5m])) +
  sum(rate(app_implicit_errors_total[5m]))
) / sum(rate(http_requests_total[5m]))

Saturation as a leading indicator

Saturation differs from the other three signals because it measures potential failure rather than actual user impact. High CPU doesn't mean users are suffering—yet. But once saturation hits critical thresholds, degradation is inevitable. Track saturation with utilization metrics (CPU, memory, disk) and queue depth metrics (connection pools, message backlogs).

Set saturation alerts at warning levels (e.g., 80%) to trigger proactive scaling or investigation before users notice. In Kubernetes environments, monitor pod resource requests versus limits, and track HPA replica counts approaching maximums. This proactive approach aligns with SLO-driven alerting principles that prioritize burn rate over static thresholds.

How do the Four Golden Signals compare to RED and USE methods?

Teams often ask whether to use Golden Signals, RED (Rate, Errors, Duration), or USE (Utilization, Saturation, Errors). The answer depends on your abstraction layer. Golden Signals are service-centric and user-focused, making them ideal for microservices and APIs. RED is essentially a subset of Golden Signals tailored specifically for request-driven services. USE is infrastructure-centric, better suited for bare-metal servers, VMs, and storage systems where "traffic" is less meaningful than resource utilization.

In practice, I use Golden Signals for every service dashboard and USE metrics for node-level dashboards. The correlation happens during incident response: high error rate (Golden Signal) triggers a drill-down into CPU saturation (USE method). This layered approach prevents over-instrumentation while ensuring no blind spots exist between application and infrastructure layers.

How do you implement Golden Signals alerting without burning out on-call engineers?

The goal of monitoring is actionable insight, not comprehensive data collection. Every alert based on the Four Golden Signals of Monitoring must pass the "so what?" test. If an alert fires and the on-call engineer cannot immediately identify a user impact or required action, delete or tune it. I follow a strict three-tier alerting strategy:

1. Critical (Page): Error budget burn rate exceeds threshold OR saturation > 95% with correlated latency/errors. Requires immediate human intervention.
2. Warning (Ticket): Saturation > 80% OR p99 latency degrading but within SLO. Investigate during business hours.
3. Info (Dashboard only): Traffic anomalies, gradual saturation increase. No notification, used for capacity planning and post-incident analysis.

Use multi-window burn rate alerts instead of static thresholds. A 5-minute window catches sudden outages; a 1-hour window catches chronic degradation. This approach dramatically reduces false positives compared to traditional "CPU > 80% for 5 minutes" alerts. For teams managing compliance frameworks like ISO 27001, document your alerting rationale and review quarterly to ensure alignment with business risk tolerance.

Practical Prometheus recording rules

Pre-compute your Golden Signals using recording rules to reduce query-time load and ensure consistency across dashboards and alerts:

groups:
- name: golden_signals
  interval: 30s
  rules:
  - record: job:http_latency:p99
    expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))
  - record: job:http_error_rate:ratio
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job)
  - record: job:saturation:cpu_ratio
    expr: avg(container_cpu_usage_seconds_total) by (job) / avg(container_spec_cpu_quota) by (job)

Building Reliable Systems with the Four Golden Signals of Monitoring

Adopting the Four Golden Signals of Monitoring transforms your observability from reactive chaos to proactive reliability. Start with one critical service, instrument all four signals properly, and define alerts tied to user impact rather than infrastructure vanity metrics. Remember that saturation predicts future pain, errors measure current pain, latency measures user experience quality, and traffic provides essential context. This framework scales from single-server deployments to multi-region Kubernetes clusters.

If your team struggles with alert fatigue, inconsistent dashboards, or compliance audit findings related to monitoring gaps, let's talk. I help engineering teams build observability systems that actually reduce MTTR and support business growth. Reach out to discuss your monitoring architecture or explore my DevOps consulting services for hands-on implementation support.