Monitoring with Prometheus and Grafana: Complete Setup (2026 Guide)

By Khimananda Oli | Last reviewed: August 2026

A server that goes down at 3 a.m. with no alert, a memory leak nobody noticed until requests started timing out, a disk that filled silently — every one of these is a monitoring gap, not bad luck. Monitoring with Prometheus and Grafana closes that gap: Prometheus scrapes metrics from your servers and services on a schedule, stores them as time series, and Grafana turns them into dashboards while Alertmanager pages you before users notice. This guide builds the whole stack end to end with Docker Compose — real prometheus.yml, node_exporter, PromQL queries, dashboards, and alert rules included. If your host is still bare, start with the initial Ubuntu server setup guide first, then come back here.

How does the Prometheus pull model work?

Most monitoring tools you have used are push-based: an agent on each host sends metrics up to a central server. Prometheus inverts this. It is a pull system — Prometheus itself reaches out and scrapes an HTTP endpoint (conventionally /metrics) on each target at a fixed scrape_interval, usually every 15 seconds. Each target exposes plain-text metrics that look like this:

node_cpu_seconds_total{cpu="0",mode="idle"} 84512.19
node_memory_MemAvailable_bytes 2.03104256e+09
node_filesystem_avail_bytes{mountpoint="/"} 1.5273984e+10

Every metric has a name and optional labels (the {key="value"} pairs) that let you slice the same metric by CPU core, mount point, HTTP status code, and so on. Prometheus timestamps each scrape and stores it in its local time-series database (TSDB). The pull model has real operational advantages:

• Service discovery is central. Prometheus knows every target because you list them (or discover them via Kubernetes, Consul, or file-based discovery) — there is no guessing which agents are alive.
• A down target is itself a signal. If a scrape fails, the synthetic up metric goes to 0, so "the exporter stopped responding" is a first-class, alertable event.
• No credentials on every host pushing outward — targets just expose a read-only endpoint on the internal network.

How do you install Prometheus and Grafana with docker-compose?

The fastest reproducible setup runs every component as a container. Create a project directory and this docker-compose.yml. It runs Prometheus, Grafana, Alertmanager, and node_exporter (which exposes host CPU, memory, disk, and network metrics):

services:
  prometheus:
    image: prom/prometheus:v3.5.0
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alert.rules.yml:/etc/prometheus/alert.rules.yml:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
    ports:
      - "9090:9090"

  node_exporter:
    image: prom/node-exporter:v1.9.1
    restart: unless-stopped
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.rootfs=/host'
    ports:
      - "9100:9100"

  alertmanager:
    image: prom/alertmanager:v0.28.1
    restart: unless-stopped
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    ports:
      - "9093:9093"

  grafana:
    image: grafana/grafana:12.1.0
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=change_me_now
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"

volumes:
  prometheus_data:
  grafana_data:

Pin image tags to explicit versions — never latest — so a redeploy cannot silently change behaviour. The named volumes (prometheus_data, grafana_data) persist your metrics history and dashboards across container restarts. In production, put this behind a reverse proxy with TLS rather than exposing ports 9090 and 3000 directly; the same hardening ideas apply as in any DevOps and cloud infrastructure engagement.

The prometheus.yml scrape configuration

The heart of the setup is prometheus.yml. It defines the global scrape interval, points Prometheus at Alertmanager, loads your alert rules, and lists the scrape_configs — one job per group of targets:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'alert.rules.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node_exporter:9100']
        labels:
          instance: 'web-01'

  - job_name: 'laravel_app'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['app:8080']

Because these run on the same Docker network, targets are addressed by service name (node_exporter:9100), not localhost. Bring the stack up with docker compose up -d, then open http://your-server:9090/targets — every job should show State: UP. Grafana lands on port 3000; log in with admin and the password you set.

What are the PromQL basics you need to read a metric?

PromQL (Prometheus Query Language) is how you ask questions of the stored data. The trick most newcomers miss: many raw metrics are counters that only ever increase (total CPU seconds, total HTTP requests), so you almost never graph them directly — you wrap them in rate() to get a per-second rate over a time window. A few building blocks cover most day-to-day queries:

# Per-core CPU usage as a percentage (100 minus idle rate)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory used as a percentage of total
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Root filesystem free space, in percent
node_filesystem_avail_bytes{mountpoint="/"}
  / node_filesystem_size_bytes{mountpoint="/"} * 100

# HTTP request rate per second, grouped by status code
sum by (status) (rate(http_requests_total[5m]))

Read a query inside-out. rate(node_cpu_seconds_total{mode="idle"}[5m]) takes the idle-CPU counter, computes its per-second increase averaged over the last 5 minutes, avg by (instance) collapses the per-core series into one number per host, and subtracting from 100 flips "idle" into "busy". The [5m] is a range vector — a window of samples — which rate() needs; leave it off and the query errors. Paste any of these into the Prometheus expression browser at /graph to see them plotted immediately.

How do you build Grafana dashboards from Prometheus data?

Grafana does not store metrics — it queries Prometheus and draws them. The setup is three steps:

1. Add the data source. In Grafana, go to Connections → Data sources → Add data source → Prometheus and set the URL to http://prometheus:9090 (the service name on the Docker network). Save and test; you should see "Successfully queried".
2. Import a ready-made dashboard. You do not have to build panels by hand. Go to Dashboards → New → Import and enter dashboard ID 1860 ("Node Exporter Full") — it renders CPU, memory, disk, and network for every host scraped by the node job. Pick your Prometheus data source when prompted.
3. Add your own panels. For application metrics, create a panel, paste a PromQL query such as sum by (status) (rate(http_requests_total[5m])), choose a visualization (time series, stat, gauge, or table), and set the unit and thresholds so a red panel means "bad" at a glance.

Group related panels onto one board per concern — one for host health, one per application — and use dashboard variables (for example a $instance drop-down populated from label_values(node_uname_info, instance)) so a single dashboard serves every server. You can see this kind of observability work in the DevOps case studies.

How do you set up alerts with Alertmanager?

Dashboards are for when you are looking; alerts are for when you are not. Prometheus evaluates alerting rules on its own evaluation_interval, and when a rule expression stays true for its for duration, Prometheus fires the alert to Alertmanager, which handles grouping, silencing, and routing to a receiver. First, the rules file (alert.rules.yml):

groups:
  - name: host-alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Target {{ $labels.instance }} is down"
          description: "{{ $labels.job }} on {{ $labels.instance }} has been unreachable for 2 minutes."

      - alert: HighCpuUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage has been above 85% for 10 minutes."

      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"

The for clause is what separates a real incident from a transient blip: HighCpuUsage only fires after CPU has stayed above 85% for a full 10 minutes, so a brief spike during a deploy never wakes you. Next, route those alerts in alertmanager.yml — this example groups by alert name and sends to a Slack webhook:

route:
  receiver: 'team-slack'
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: 'team-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#alerts'
        send_resolved: true
        title: '{{ .CommonAnnotations.summary }}'

send_resolved: true makes Alertmanager also post when the condition clears, so the channel tells the whole story — fired and recovered. Store the real webhook URL in a secret or environment variable rather than committing it, exactly as you would with any pipeline secret in a GitLab CI/CD pipeline.

What should you monitor first, and what are common mistakes?

Do not try to graph everything on day one. Start with the signals that predict user-facing pain, then expand. A sensible first pass:

• The four host basics: CPU, memory, disk space, and disk I/O — node_exporter gives you all of these out of the box.
• The up metric for every target, so a dead exporter or service pages you immediately.
• Application golden signals: request rate, error rate, and latency, if your app exposes a /metrics endpoint (a client library or an exporter provides this).

The mistakes that bite teams later are avoidable: setting scrape_interval too aggressively (1s scrapes balloon storage for little insight — 15s is the sane default); ignoring cardinality (a label like user_id creates a new time series per user and can overwhelm the TSDB); alerting without a for duration (every transient spike becomes noise you will start ignoring); and forgetting --storage.tsdb.retention.time, so the disk quietly fills. Keep labels bounded, alert on symptoms not causes, and treat your monitoring config as version-controlled code.

Conclusion

You now have a complete Prometheus and Grafana monitoring stack: a pull-based scrape model feeding a time-series database, node_exporter reporting host health, PromQL turning raw counters into rates, Grafana dashboards for the human eye, and Alertmanager to page you when a rule stays broken. Start by bringing up the docker-compose stack and importing dashboard 1860 today, then add one alert rule per real failure mode you have actually hit. If you want this monitoring set up, tuned, and wired into your on-call rotation for your team, get in touch or explore my DevOps and cloud services to see how it fits your infrastructure.