Rook: Ceph on Kubernetes

By Khimananda Oli | Last reviewed: August 2026

Stateful workloads on Kubernetes fail without reliable persistent storage, and cloud-native teams increasingly turn to Rook: Ceph on Kubernetes to solve this. Unlike external storage arrays or basic hostPath volumes, Rook automates the deployment, bootstrapping, and management of a full Ceph storage cluster directly inside your K8s environment. This guide walks you through a production-ready implementation based on real-world deployments I have managed across bare-metal and hybrid environments.

How does Rook: Ceph on Kubernetes architecture work?

Understanding the architecture prevents costly misconfigurations during initial setup. Rook operates as two distinct layers: the Rook Operator and the Ceph Daemons. The operator is a lightweight controller that watches for Custom Resources (CRDs) like CephCluster, CephBlockPool, and CephFilesystem. When you apply these manifests, the operator translates your desired state into actual Kubernetes resources—Deployments, DaemonSets, Services, and ConfigMaps—that run the Ceph daemons.

The critical distinction here is separation of concerns. The operator never handles data; it only orchestrates. Your application pods interact solely with Ceph through the CSI driver, completely unaware of Rook's existence. This means if the operator pod crashes temporarily, existing volumes remain fully functional. For teams evaluating alternatives, I compare this pattern against simpler options in my Longhorn distributed storage guide, but Rook’s maturity makes it the default choice for complex stateful workloads requiring POSIX compliance or S3 compatibility.

How do you install and configure Rook: Ceph on Kubernetes?

Installation requires careful sequencing. Skipping prerequisites causes silent failures during OSD bootstrap. Always verify your nodes have raw, unformatted disks dedicated to Ceph before proceeding.

Step 1: Deploy the Rook Operator

Add the official Helm repository and install the operator. Using Helm simplifies future upgrades significantly compared to raw manifests.

helm repo add rook-release https://charts.rook.io/release
helm repo update

helm install --create-namespace --namespace rook-ceph \
  rook-ceph rook-release/rook-ceph \
  --set csi.cephFSPlugin.enabled=true \
  --set csi.rbdPlugin.enabled=true \
  --version v1.16.0

Wait for the operator pod to reach Ready status. Check logs immediately for RBAC errors or missing CRD registrations:

kubectl -n rook-ceph rollout status deployment/rook-ceph-operator
kubectl -n rook-ceph logs -l app=rook-ceph-operator --tail=100

Step 2: Create the CephCluster Resource

This is where most production issues originate. Below is a minimal viable configuration for a three-node cluster with dedicated OSD devices. Never use useAllDevices: true in production—it risks consuming system disks.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.0
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
    modules:
      - name: dashboard
        enabled: true
  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
      - name: node-01
        devices:
          - name: sdb
      - name: node-02
        devices:
          - name: sdb
      - name: node-03
        devices:
          - name: sdb
  resources:
    osd:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        memory: "4Gi"

Apply this manifest and monitor progress. The operator creates MON pods first, then MGR, then OSDs. Full cluster convergence typically takes 5–15 minutes depending on disk speed and network latency.

Step 3: Define Storage Classes

Create a CephBlockPool and corresponding StorageClass for RBD volumes. This enables standard PVC claims from applications.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  pool: replicapool
  clusterID: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

If you need shared filesystem access for multi-pod read-write scenarios (like WordPress or Jenkins), create a CephFilesystem resource separately. Understanding when to use block versus file storage aligns with broader Kubernetes persistent volume best practices.

What are the common production pitfalls with Rook Ceph?

After managing multiple Rook clusters in audit-sensitive environments, certain failure modes recur consistently. Addressing these proactively prevents 3 AM pages.

• Insufficient MON quorum: Running fewer than three monitors risks split-brain during node maintenance. Always deploy odd numbers (3 or 5).
• Missing resource limits: Unbounded OSD pods can consume all node memory during rebalancing, triggering OOM kills. Always set explicit requests and limits as shown above.
• Ignoring PG autoscaling: Manual placement group tuning is obsolete. Enable pg_autoscale_mode: on in pool specs to let Ceph adjust dynamically.
• No monitoring integration: Ceph exposes rich Prometheus metrics at :9283/metrics. Without dashboards, you cannot detect degradation before user impact. See my Prometheus and Grafana monitoring stack guide for integration patterns.
• Skipping backup validation: Rook manages storage, not backups. Implement regular snapshot schedules and test restores quarterly. Compliance frameworks like SOC 2 require evidence of recoverability.

How does Rook Ceph compare to other Kubernetes storage options?

Choosing storage involves trade-offs between complexity, performance, and operational overhead. This comparison reflects 2026 production realities, not marketing claims.

In practice, I recommend Rook when you need object storage alongside block volumes, operate on bare metal, or require POSIX-compliant shared filesystems. For teams running exclusively in AWS/Azure/GCP with simple database workloads, native CSI drivers reduce operational burden significantly. Longhorn fills a niche for smaller clusters needing straightforward replication without Ceph’s learning curve. Evaluate your actual protocol requirements before defaulting to any single solution.

How do you monitor and maintain a Rook Ceph cluster?

Day-2 operations determine long-term success. Rook exposes comprehensive telemetry, but raw metrics are useless without context.

1. Enable the Ceph Dashboard: Already activated in the cluster spec above. Access it via port-forward or Ingress. Use it for visual health checks, not automation.
2. Configure Prometheus ServiceMonitor: Rook ships a ready-to-use ServiceMonitor. Apply it to enable automatic metric scraping. Key alerts include CephHealthErr, CephOSDNearFull, and CephMonQuorumLost.
3. Implement log aggregation: Ceph logs are verbose. Forward them to your centralized logging system with structured parsing. My structured logging guide covers filtering noise effectively.
4. Schedule regular health checks: Automate ceph status and ceph df outputs weekly. Track trends in PG states and OSD utilization over time.
5. Test upgrade paths in staging: Never upgrade Ceph versions directly in production. Validate compatibility with your specific workload patterns first.

Next Steps for Production Rook Deployments

Deploying Rook: Ceph on Kubernetes successfully requires treating it as a platform component, not an afterthought. Start with the configuration patterns above, validate thoroughly in non-production, and integrate monitoring before onboarding real workloads. Document your recovery procedures and test them regularly—compliance auditors and incident responders will thank you. If your team needs hands-on support designing or hardening a Rook deployment, especially for regulated environments, reach out to discuss your specific requirements.