Rotate encryption keys

ForPlatform operatorSecurity engineer

LevelAdvanced

This guide covers two independent keys used by crypto-shredding:

Key	Environment variable	Purpose
Platform KEK	`PLATFORM_KEK`	Wraps each subject DEK in the dedicated subject-keys database.
Blind-index key	`KUMIKO_BLIND_INDEX_KEY`	Creates HMAC fingerprints for equality lookups on lookupable encrypted fields.

The subject DEK and the encrypted application data do not change during a platform-KEK rotation. Only the wrapper around the DEK changes.

Before either rotation:

Confirm the current and new values are stored in your password manager.
Take and verify a backup according to your deployment runbook.
Prepare a rollback or break-glass path before changing production.
Never print key values, put them in Git, or place a management token in an app pod.
Keep the old KEK until the backup-retention window has passed and the new generation has been verified.

For a production incident, follow your deployment team’s backup, Key Manager, and break-glass procedure. The API facts below are checked against the current framework implementation; the cluster-specific runbook is intentionally kept in the private operations repository.

Rotate the platform KEK

The adapter records a kek_version on every wrapped subject-key row. During a rotation, the active adapter reads the current generation and the configured older generations, while new rows use the active generation.

Use this order:

Back up the dedicated subject-keys database.
Generate and store the new 32-byte base64 KEK:
Terminal window
```
openssl rand -base64 32
```
Roll out the app with the new active KEK and the old KEK in previousKeks. The active version must be higher than the old version. When the app reads this configuration from environment variables, set the matching PLATFORM_KEK_VERSION, PLATFORM_KEK_PREVIOUS, and PLATFORM_KEK_PREVIOUS_VERSION values too:
```
createPgKmsAdapter({
  databaseUrl: env.SUBJECT_KEYS_DATABASE_URL,
  platformKek: NEW_KEK,
  kekVersion: 2,
  previousKeks: { 1: OLD_KEK },
});
```
Wait for the rollout and verify that the app can still read an existing encrypted record. Do not re-wrap rows while an old pod can still create generation-1 rows.
Run a dry-run of the re-wrap. The function scans live and erased rows, skips erased tombstones, and reports failures without writing in dry-run mode:
```
import { rewrapSubjectKeys } from "@cosmicdrift/kumiko-framework/crypto";

const result = await rewrapSubjectKeys({
  databaseUrl: SUBJECT_KEYS_URL,
  fromKeks: { 1: OLD_KEK },
  toKek: NEW_KEK,
  toKekVersion: 2,
  dryRun: true,
});
```
In a shared subject-keys table, rows belonging to other apps may appear as authentication failures because their old KEKs are different. Compare the failures with the subjects that belong to this app. Do not treat an unexplained failure as harmless.

Run the same operation without dryRun after reviewing scanned, rewrapped, skippedErased, and failures:

const result = await rewrapSubjectKeys({
  databaseUrl: SUBJECT_KEYS_URL,
  fromKeks: { 1: OLD_KEK },
  toKek: NEW_KEK,
  toKekVersion: 2,
});

Verify the result. The table should contain generation-2 rows for the subjects owned by this app:
```
SELECT kek_version, count(*)
FROM kumiko_subject_keys
WHERE cipher_key IS NOT NULL
GROUP BY kek_version;
```
Stronger verification uses an adapter configured with only the new KEK and then reads representative subjects. Also perform a login and PII-detail smoke test.
Remove previousKeks and its environment values only after verification and the backup-retention window. Roll out the change and keep the old KEK in protected storage until that point.

Blind-index values are derived projection data. The key has no generation window, so rotation creates a short interval in which old encrypted rows may not match equality queries.

Generate and securely store a new key:
Terminal window
```
openssl rand -base64 32
```
Deploy the app with the new KUMIKO_BLIND_INDEX_KEY.
Immediately rebuild every entity projection that contains lookupable fields. A deployment’s kumiko schema apply can run queued rebuilds. For a manual rebuild, use rebuildProjection through the app’s current registry, or the jobs feature’s jobs:job:projection-rebuild job. The rebuild must run in a process that has the new blind-index key configured.
Keep the old key available until the rebuild and login smoke test succeed.
Verify that lookupable encrypted rows have populated <field>_bidx values and that login, password reset, and other equality lookups work.

The schema apply init container must receive KUMIKO_BLIND_INDEX_KEY too. The framework deliberately stops a rebuild when populated blind-index data exists but the rebuild process has no key.

If something fails

Stop the rollout if a new adapter cannot unwrap a representative subject.
Restore the previous active configuration while the old KEK is still available.
Do not delete or overwrite the old subject-keys backup.
If a blind-index rebuild fails, leave the live projection in place, fix the key wiring, and retry the rebuild.
Record the failure and the recovery result in the operational change log.

Rotate encryption keys

Rotate the platform KEK

Rotate the blind-index key

If something fails

See also