Encrypt etcd Secrets at Rest

August 14, 2026 Khimananda Oli 9 min read Database

By Khimananda Oli | Last reviewed: August 2026

Kubernetes stores all cluster state, including Secrets, as plaintext objects in etcd by default. If an attacker gains read access to the etcd data directory or a backup snapshot, they can extract every credential, TLS certificate, and API token without touching your application layer. To encrypt etcd secrets at rest, you must configure the API server’s EncryptionConfiguration resource so that sensitive data is written to disk in ciphertext rather than raw JSON. This guide walks through the exact configuration, key management, and verification steps required for production clusters in 2026.

Quick answer: To encrypt etcd secrets at rest, create an EncryptionConfiguration file specifying a KMS or AES-CBC provider, mount it into the kube-apiserver pod or static manifest, and restart the API server. Existing secrets remain plaintext until rewritten; run a re-encryption job after enabling to ensure all data is encrypted on disk.

Unencrypted vs encrypted etcd secrets at rest: enabling EncryptionConfiguration routes writes through a KMS or local cipher before persisting to etcd.

Why should you encrypt etcd secrets at rest in Kubernetes?

Etcd is the single source of truth for your entire cluster. Without encryption at rest, anyone with filesystem access—whether through a compromised node, a leaked backup, or a misconfigured cloud storage bucket—can read every Secret in cleartext. In my work with SOC 2 and ISO 27001 audits across Nepal-based fintechs and global SaaS platforms, this is consistently the first finding auditors check. Compliance frameworks explicitly require cryptographic protection for stored credentials, and "we trust our infrastructure team" does not satisfy evidence requirements.

Beyond compliance, defense-in-depth demands this control. Network policies, RBAC, and pod security standards protect the API surface, but they do nothing if the underlying datastore is exposed. As discussed in Kubernetes secrets management done right, application-layer secret injection tools like Vault or External Secrets Operator complement but do not replace datastore encryption. You need both: external tools prevent secrets from living in manifests, while etcd encryption protects whatever still lands in the database.

How do you configure EncryptionConfiguration to encrypt etcd secrets at rest?

The EncryptionConfiguration is a structured YAML file read by the kube-apiserver at startup. It defines an ordered list of providers for each resource type. The API server uses the first provider for encryption and tries all listed providers in order for decryption, which enables safe key rotation.

Create the configuration file

Save this to /etc/kubernetes/enc/enc-config.yaml on each control plane node:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          name: aws-kms-provider
          endpoint: unix:///var/run/kms-plugin/socket.sock
          cachesize: 1000
          timeout: 3s
      - identity: {}

The kms provider delegates envelope encryption to an external key management service via a gRPC plugin. The identity provider performs no encryption and exists solely as a fallback during rotation. Never place aescbc or secretbox above kms unless you are intentionally migrating away from managed keys.

Mount and reference the file in the API server

For kubeadm-managed clusters, edit /etc/kubernetes/manifests/kube-apiserver.yaml:

Add a volume: - name: enc-config\n hostPath:\n path: /etc/kubernetes/enc\n type: DirectoryOrCreate
Add a volumeMount to the container: - name: enc-config\n mountPath: /etc/kubernetes/enc\n readOnly: true
Add the flag: --encryption-provider-config=/etc/kubernetes/enc/enc-config.yaml
If using KMS v2 (recommended for 2026), also add: --encryption-provider-config-automatic-reload=true to allow config changes without full restart.

The kubelet watches the static manifest directory and restarts the API server pod automatically when the file changes. Expect 30–90 seconds of API unavailability during this restart; schedule it during a maintenance window.

Envelope encryption sequence: the API server generates a local DEK, wraps it via KMS, and stores only the wrapped key alongside ciphertext in etcd.

Which encryption provider should you choose for etcd?

Kubernetes supports several providers, but only two are appropriate for production in 2026. The choice depends on your operational maturity, compliance requirements, and whether you can manage key lifecycle externally.

Provider	Key Storage	Audit Trail	Rotation Complexity	Best For
KMS v2 (Recommended)	External (AWS KMS, GCP Cloud KMS, Azure Key Vault)	Yes — cloud provider logs every key operation	Low — rotate KEK in cloud console; API server auto-reloads	Production, SOC 2/ISO 27001, multi-team clusters
aescbc	Local file on control plane nodes	No — manual log correlation required	High — requires config edit + full re-encryption + restart	Air-gapped environments, labs, legacy clusters pre-v1.29
identity	N/A (plaintext)	N/A	N/A	Decryption fallback during rotation only
secretbox	Local file	No	High	Deprecated for new deployments; use KMS v2 instead

In practice, I recommend KMS v2 for every cluster that has internet connectivity and runs on a major cloud or hybrid platform with HSM-backed key management. The audit trail alone satisfies most compliance evidence requests without custom logging pipelines. Reserve aescbc for genuinely air-gapped government or industrial environments where external KMS is impossible—and pair it with rigorous key distribution procedures documented in your ISMS.

How do you rotate encryption keys without downtime?

Key rotation is where most teams break their clusters. The critical rule: never remove a provider that still holds live encrypted data. The API server decrypts using the first matching provider in the list, so you must maintain backward compatibility until every object has been re-encrypted.

Safe rotation procedure

Add the new key as the first provider. Keep the old key second. The API server now encrypts new writes with the new key but can still decrypt old objects.
Re-encrypt all existing secrets. Run: kubectl get secrets --all-namespaces -o json | kubectl replace -f -. This reads each secret (decrypting with any available provider) and writes it back (encrypting with the new primary provider). For large clusters, batch this by namespace to avoid API throttling.
Verify ciphertext. Use etcdctl get /registry/secrets/default/my-secret --print-value-only | hexdump -C to confirm the stored value begins with the new provider’s prefix (e.g., k8s:enc:kms:v2:aws-kms-provider:).
Remove the old provider from the configuration only after verification completes across all namespaces.
Restart the API server if automatic reload is disabled.

This process is identical whether you rotate a cloud KEK version or swap local AES keys. The difference is that cloud KMS rotations often happen transparently on the provider side; you still need to trigger re-encryption in Kubernetes because the DEK wrapping metadata must update.

How do you verify that secrets are actually encrypted on disk?

Configuration alone is not proof. Auditors and security teams need binary evidence that plaintext no longer exists in etcd. This verification step is non-negotiable for Kubernetes RBAC and security hardening attestations.

Direct etcd inspection

# Access etcd with proper certs
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-db-password \
  --print-value-only | strings | head -5

If encryption is active, output will show binary gibberish and a provider prefix like k8s:enc:kms:v2:. If you see readable JSON with "password":"...", encryption is not applied to that object. Re-run the re-encryption command and check again.

Automated compliance checks

For continuous validation, integrate etcd encryption checks into your monitoring stack. A simple cron job that samples random secrets and validates the ciphertext prefix catches configuration drift before auditors do. Pair this with Prometheus metrics monitoring fundamentals to alert on API server encryption errors or KMS latency spikes that could indicate plugin failures.

Provider comparison for encrypt etcd secrets at rest: KMS v2 offers auditability and managed keys, while aescbc suits isolated environments despite higher operational burden.

Common mistakes when implementing etcd encryption

After reviewing dozens of cluster configurations, these errors appear repeatedly:

Forgetting to re-encrypt existing secrets. Enabling encryption only affects new writes. Old secrets remain plaintext indefinitely. Always run the replacement command immediately after configuration.
Storing encryption keys on the same disk as etcd data. With aescbc, if your key file lives in /etc/kubernetes/enc/ on the same volume as /var/lib/etcd/, a single disk compromise exposes everything. Use separate volumes with strict permissions (0600, owned by root).
Not testing decryption after KMS outages. If your cloud KMS becomes unreachable, the API server cannot decrypt existing secrets. Configure health checks and have a disaster recovery runbook that includes temporary identity-provider fallback for emergency access.
Ignoring backup encryption. Etcd snapshots contain the same data as the live store. Ensure your backup toolchain (Velero, etcdctl snapshot, cloud-native backup) also encrypts artifacts at rest, or your encryption effort is bypassed entirely.

Next steps for securing your Kubernetes datastore

Encrypting etcd secrets at rest is foundational, but it is one layer in a broader security posture. Combine it with RBAC least privilege, network policies, pod security standards, and external secret stores to achieve genuine defense-in-depth. Document your encryption configuration, key rotation schedule, and verification procedures as part of your compliance evidence package—auditors will ask for them.

If you need help designing a compliant Kubernetes security architecture or validating your existing etcd encryption setup, reach out to discuss your cluster security requirements. I work with teams across Nepal and globally to build infrastructure that passes audits and survives incidents.

Frequently Asked Questions

It prevents attackers with direct disk or backup access from reading Kubernetes secrets in plaintext. Without encryption, anyone obtaining an etcd snapshot can extract credentials, tokens, and certificates immediately. This control mitigates physical theft, storage compromise, and unauthorized backup restoration risks specifically targeting the cluster state database.

Use aescbc or secretbox as the primary encryption provider for etcd in 2026. Secretbox offers better performance with XSalsa20-Poly1305, while aescbc provides AES-CBC with PKCS7 padding. Avoid deprecated providers like aesgcm for new deployments due to nonce reuse risks during high-write workloads on active clusters.

Run etcdctl get with hexdump on a known secret key to confirm ciphertext output. Plaintext values indicate misconfiguration. Also check kube-apiserver logs for encryption config load success and monitor metrics like apiserver_envelope_encryption_dek_generation_failures_total to ensure the encryption envelope functions properly without silent failures.

No, you can enable encryption without downtime by updating the kube-apiserver encryption config and restarting API servers rolling. Existing secrets remain readable via old keys. New writes use the new provider. Plan a subsequent read-decrypt-rewrite cycle to re-encrypt all existing secrets under the current key.

Losing the encryption key renders all encrypted secrets permanently unrecoverable. The API server cannot decrypt stored data, causing pod failures and authentication breakdowns. Always store keys in a dedicated KMS or HSM with strict access controls and verified backup procedures separate from the etcd storage volume itself.

Yes, Kubernetes supports external KMS providers via the kms plugin interface for etcd encryption. Configure the EncryptionConfiguration with a kms provider pointing to your cloud KMS endpoint. This delegates key management to the cloud provider, enabling automatic rotation and audit logging while keeping DEKs encrypted externally.

Rotate etcd encryption keys at least annually or after any suspected compromise. After adding a new key as primary, trigger a full secret rewrite to re-encrypt all data. Then demote and eventually remove old keys only after confirming all secrets have been successfully migrated to the new key version.

Expect five to ten percent increased latency on secret read and write operations. Secretbox typically outperforms aescbc due to simpler cryptographic primitives. Monitor etcd request duration histograms and API server response times after enabling encryption to quantify actual overhead in your specific workload and hardware environment.

Add the new encryption provider as first in the config list while retaining identity as fallback. Restart API servers, then run kubectl get secrets --all-namespaces -o json | kubectl replace -f - to force rewrites. Verify ciphertext in etcd, then remove the identity provider once migration completes successfully.

No, etcd encryption applies cluster-wide using a single EncryptionConfiguration. All secrets share the same key set regardless of namespace. For per-namespace isolation, use external secret stores like Vault or Sealed Secrets that encrypt before submission, keeping etcd encryption as a secondary defense layer only.

Misordering providers in EncryptionConfiguration causes reads to skip decryption. Missing identity provider during migration locks out unencrypted secrets. Incorrect base64 encoding of keys triggers API server startup failures. Always validate config syntax with kube-apiserver dry-run and test secret retrieval immediately after applying changes to catch errors early.

Etcd encryption satisfies technical controls for data-at-rest protection but alone does not guarantee SOC2 compliance. Auditors also require key management policies, access logging, rotation evidence, and incident response procedures. Combine encryption with comprehensive governance documentation and regular access reviews to meet full compliance obligations effectively.

Velero backs up Kubernetes API objects, not raw etcd data, so exported secrets are decrypted during retrieval. Restored secrets get re-encrypted automatically by the target cluster’s current encryption config. Raw etcd snapshots taken outside Velero retain their original encryption state and require matching keys for restoration.

Existing secrets remain unencrypted until rewritten. Adding an encryption provider only affects new writes. You must manually trigger updates on all secrets to re-encrypt them. Confirm the encryption config lists your provider first and includes identity as fallback during migration to avoid accidental lockouts during this process.

Use kube-bench or custom scripts calling etcdctl with hexdump to verify ciphertext in automated tests. Integrate checks into GitOps pipelines that validate EncryptionConfiguration syntax before deployment. Tools like Polaris or Datree can enforce encryption policy compliance as part of pre-commit hooks and continuous security scanning workflows.