Load encryption keys from a key manager

This guide moves three existing Kumiko keys out of the deployment environment and into Scaleway Key Manager. The deployment keeps a ciphertext and a revocable decrypt token. Each pod decrypts the keys once during boot and holds the plaintext only in memory.

This procedure keeps each key’s value unchanged. Existing encrypted data should remain readable after the switch.

Before you start

Complete this procedure with another operator. You need:

• a working Kumiko production app
• a Scaleway account and project
• permission to manage Scaleway Key Manager keys
• permission to create a separate decrypt-only application token
• access to the app’s Pulumi stack and Kubernetes deployment
• curl and jq in your terminal
• the current plaintext keys in an approved password manager
• a tested way to deploy the previous infrastructure revision
• an application probe that reads data written before this change

Check the local tools:

curl --version
jq --version

Choose concrete values for the examples:

export APP_NAME="YOUR_APP_NAME"
export KMS_REGION="fr-par"
export KMS_KEY_ID="YOUR_BARE_KEY_UUID"
export KMS_DECRYPT_TOKEN="YOUR_DECRYPT_ONLY_TOKEN"
export KMS_MANAGEMENT_TOKEN="YOUR_MANAGEMENT_TOKEN"

Use the bare key UUID for KMS_KEY_ID. Do not include fr-par/.

Stop before handling plaintext

• Never commit a plaintext key, API token, .env, or temporary key file.
• Never pass the management token to a pod. A pod needs decrypt permission only.
• Do not paste keys into chat, tickets, CI variables, or command arguments.
• Keep a separate offline recovery copy of every plaintext key.
• Test the plaintext rollback path before removing the old deployment value.

What the three keys protect

A key is a random secret that makes encrypted data readable again. Kumiko uses three app-wide keys for different jobs:

Key	Plaintext variable	Ciphertext variable	Purpose
Platform KEK	PLATFORM_KEK	PLATFORM_KEK_CIPHERTEXT	Encrypts each person’s data-encryption key in the subject-key store
Blind-index key	KUMIKO_BLIND_INDEX_KEY	KUMIKO_BLIND_INDEX_KEY_CIPHERTEXT	Creates exact-match fingerprints for encrypted fields
Secrets master key	KUMIKO_SECRETS_MASTER_KEY_V1	KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT	Encrypts tenant secrets, encrypted configuration, and MFA secrets

All three values are 32 random bytes encoded as base64. Losing the platform KEK or secrets master key can make protected data unreadable. Losing the blind-index key breaks exact-match lookups until the indexes are rebuilt.

This guide wraps existing values. Key rotation changes a key’s value and is a separate operation.

What this protects

A compromised live pod still has enough information to ask Key Manager for the plaintext. This setup does not protect a fully compromised running process.

It does reduce exposure elsewhere:

• Pulumi state, Kubernetes Secret backups, CI logs, and old deployment exports contain ciphertext instead of the key.
• The decrypt token can be revoked. A copied plaintext key cannot.
• Key Manager records decrypt calls in its audit log.
• Hosting encrypted data and the wrapping key with different providers separates their custody.

Running pods keep working during a Key Manager outage. New pods cannot start until decrypt works again or you use the plaintext recovery path.

1. Create separate management and decrypt credentials

Create two Scaleway API credentials:

1. A management credential for Pulumi and the setup operator. It may create and protect Key Manager keys.
2. An application credential with only KeyManagerKeyDecrypt for the required project and key.

Store both credentials in the approved secret store. Put only the decrypt token in the app deployment.

What should happen

The application credential can call decrypt on the chosen key. It cannot create, change, rotate, or delete a Key Manager key.

Do not continue with one shared token

A management credential can delete the key container. Deleting it can make every wrapped key for the app permanently unavailable. Create and test the restricted credential before proceeding.

Common errors

• HTTP 401: the token is invalid or expired.
• HTTP 403: the token lacks decrypt permission or belongs to the wrong project.
• A decrypt token can manage keys: its policy is too broad; fix the policy before deployment.

2. Create one protected key per app

Use one Key Manager key for each app. A separate key limits the effect of an accidental deletion or compromised credential.

In the operators Pulumi stack, use the existing key pattern:

const key = new scaleway.keymanager.Key(
  "platform-kek-YOUR_APP_NAME",
  {
    name: "platform-kek-YOUR_APP_NAME",
    region: "fr-par",
    projectId,
    usage: "symmetric_encryption",
    algorithm: "aes_256_gcm",
    origin: "scaleway_kms",
    unprotected: false,
  },
  { provider, protect: true },
);

Use origin: "scaleway_kms" so Scaleway creates and retains the wrapping key. unprotected: false protects it at Scaleway; Pulumi’s protect: true adds a separate deletion guard in infrastructure state.

Do not configure scheduled rotation for this migration. A Key Manager rotation changes the version used by future encrypt calls; it does not rewrap the ciphertexts you already stored.

What should happen

After pulumi preview and reviewed pulumi up, Scaleway shows an enabled AES-256-GCM key in fr-par. Pulumi must not propose changes to any app Deployment at this step.

Get the bare key ID

The Scaleway Pulumi resource ID has the form fr-par/<uuid>. The REST endpoint already contains the region, so the app needs only the UUID:

export function bareKeyId(resourceId: string): string {
  return resourceId.split("/").pop() ?? resourceId;
}

An ID that still starts with fr-par/ produces a request path with the region twice and returns HTTP 404.

3. Prepare and test plaintext recovery

Before wrapping anything:

1. Confirm that every current plaintext key is present in the password manager.
2. Label each value with the app name, variable name, and key version.
3. Confirm that the infrastructure has an either/or branch for plaintext and ciphertext.
4. Test deploying the plaintext branch in a non-production environment or a controlled drill.
5. Record the previous working infrastructure revision and app image SHA.

The source shape should make plaintext and ciphertext mutually exclusive:

type KeySource =
  | { readonly plaintext: pulumi.Input<string> }
  | { readonly ciphertext: pulumi.Input<string> };

The concrete infrastructure helper should emit either NAME or NAME_CIPHERTEXT. A leftover plaintext value always wins and silently bypasses Key Manager.

What should happen

A reviewed recovery procedure can restore all three plaintext variables and start a pod without a Key Manager call. Do not remove the existing production values until this has been demonstrated.

4. Confirm the app recognizes every ciphertext slot

The standard secrets feature already marks KUMIKO_SECRETS_MASTER_KEY_V1 as a Key Manager slot. A custom field must carry the same metadata:

KUMIKO_SECRETS_MASTER_KEY_V1: base64Key32
  .describe("AES-256 master key for tenant-secret encryption.")
  .meta({ kumiko: { kms: true } }),

composeEnvSchema adds the matching _CIPHERTEXT field. runProdApp calls resolvePlatformKeks for every slot returned by kmsSlotsOf(schema) before boot-time consumers use the parsed environment.

Apps with a second typed env schema

Some apps parse a hand-built schema before calling runProdApp. Zod removes undeclared fields. In that setup, declare the ciphertext twin in the hand-built schema too:

KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: z.string().min(1).optional(),

Keep both schemas synchronized with a test:

test.each([...kmsSlotsOf(appComposedEnv.schema)])(
  "%s has a declared ciphertext twin",
  (slot) => {
    expect(Object.keys(appEnvSchema.shape)).toContain(`${slot}_CIPHERTEXT`);
  },
);

Apps that build crypto providers before runProdApp

Resolve once near the start of the entry point and pass the returned environment to every early consumer:

const resolvedEnv = {
  ...env,
  ...(await resolvePlatformKeks(env, {
    logPrefix: "[YOUR_APP_NAME]",
    slots: kmsSlotsOf(appComposedEnv.schema),
  })),
};

The resolver does not mutate process.env. Search for createEnvMasterKeyProvider, createEnvelopeCipher, and MasterKeyProvider to find code that may still read the unresolved environment.

What should happen

Typecheck and boot tests pass with ciphertext fields present and the plaintext slots absent. A missing slot must fail during boot, before the first login or data read.

5. Wrap one key and verify the round trip

Handle one key at a time. The example reads the plaintext without echoing it, checks that it is a base64-encoded 32-byte value, encrypts it, decrypts the new ciphertext, and compares the exact API values.

set -euo pipefail

API="https://api.scaleway.com/key-manager/v1alpha1/regions/${KMS_REGION}/keys"

read -rsp "Paste the existing plaintext key: " PLAINTEXT_KEY
printf '\n'

DECODED_BYTES=$(printf '%s' "$PLAINTEXT_KEY" | base64 -d | wc -c | tr -d ' ')
[ "$DECODED_BYTES" = "32" ] || {
  echo "Expected a base64-encoded 32-byte key; got ${DECODED_BYTES} decoded bytes" >&2
  exit 1
}

ENCRYPT_RESPONSE=$(jq -n --arg plaintext "$PLAINTEXT_KEY" '{plaintext:$plaintext}' \
  | curl --fail-with-body --silent --show-error \
      --request POST \
      --header "X-Auth-Token: ${KMS_MANAGEMENT_TOKEN}" \
      --header "Content-Type: application/json" \
      --data @- \
      "${API}/${KMS_KEY_ID}/encrypt")

CIPHERTEXT=$(printf '%s' "$ENCRYPT_RESPONSE" | jq -er '.ciphertext')

DECRYPT_RESPONSE=$(jq -n --arg ciphertext "$CIPHERTEXT" '{ciphertext:$ciphertext}' \
  | curl --fail-with-body --silent --show-error \
      --request POST \
      --header "X-Auth-Token: ${KMS_DECRYPT_TOKEN}" \
      --header "Content-Type: application/json" \
      --data @- \
      "${API}/${KMS_KEY_ID}/decrypt")

ROUNDTRIP=$(printf '%s' "$DECRYPT_RESPONSE" | jq -er '.plaintext')
[ "$ROUNDTRIP" = "$PLAINTEXT_KEY" ] || {
  echo "Round-trip mismatch; do not deploy this ciphertext" >&2
  exit 1
}

printf 'Round trip verified. Ciphertext follows; store it in the deployment config:\n%s\n' "$CIPHERTEXT"
unset PLAINTEXT_KEY ROUNDTRIP ENCRYPT_RESPONSE DECRYPT_RESPONSE

On macOS, the built-in base64 accepts -d. If your implementation requires --decode, use that spelling without changing the rest of the check.

What should happen

The script prints Round trip verified followed by one ciphertext. It never prints the plaintext key or either token.

Store the ciphertext under the app’s established Pulumi config name. A ciphertext does not need Pulumi secret encryption, but the decrypt token does:

pulumi config set YOUR_APP_CIPHERTEXT_CONFIG "$CIPHERTEXT"
pulumi config set --secret kekDecryptToken "$KMS_DECRYPT_TOKEN"
unset CIPHERTEXT

Repeat this step for:

1. PLATFORM_KEK
2. KUMIKO_BLIND_INDEX_KEY
3. KUMIKO_SECRETS_MASTER_KEY_V1

Common errors

• HTTP 401 or 403: check the token and its scope. Do not fall back to the management token in the pod.
• HTTP 404: KMS_KEY_ID probably contains fr-par/, or the key is in another region.
• jq reports a missing field: inspect the HTTP error body; do not store null or an empty string.
• Decoded length is not 32: you selected the wrong secret or encoded it twice.
• Round-trip mismatch: stop. Keep the plaintext deployment unchanged and investigate.

6. Add the decrypt-only deployment wiring

Every ciphertext slot shares these variables:

Variable	Value
PLATFORM_KEK_KMS_KEY_ID	bare Key Manager UUID
PLATFORM_KEK_KMS_TOKEN	decrypt-only token
PLATFORM_KEK_KMS_REGION	optional; defaults to fr-par

For the subject-key pair, use the existing infrastructure union:

subjectKeysKms: {
  platformKekCiphertext: config.require("YOUR_APP_PLATFORM_KEK_CIPHERTEXT"),
  kms: {
    keyId: platformKekKeyId("YOUR_APP_NAME"),
    token: config.requireSecret("kekDecryptToken"),
  },
  blindIndexKeyCiphertext: config.require("YOUR_APP_BLIND_INDEX_CIPHERTEXT"),
},

Pass the secrets master key through its ciphertext twin:

{
  KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: config.require(
    "YOUR_APP_SECRETS_MASTER_KEY_CIPHERTEXT",
  ),
}

Remove the matching plaintext fields from the deployment branch. Do not leave both forms set.

What should happen

pulumi preview shows ciphertext configuration and the decrypt token reaching the app Secret. It must not show the management token or any plaintext key.

Stop if preview removes unrelated secrets, changes another app’s key ID, or replaces the Key Manager key.

7. Roll out one app and verify it

Deploy only one app. Wait for the rollout before changing another:

kubectl -n YOUR_NAMESPACE rollout status \
  deployment/YOUR_DEPLOYMENT \
  --timeout=300s

Read the app container log:

kubectl -n YOUR_NAMESPACE logs \
  deployment/YOUR_DEPLOYMENT \
  --container YOUR_DEPLOYMENT \
  --tail=200 \
  | grep 'source='

Expected lines for each migrated slot contain:

source=key-manager keyId=YOUR_BARE_KEY_UUID region=fr-par

Any source=plaintext-env line means that a plaintext value still wins. Remove the duplicate and redeploy before calling the migration complete.

Verify old data

A green pod proves that a well-formed key loaded. It does not prove that the key matches the existing ciphertext. Run a read that uses data written before the migration:

• read an encrypted personal-data field,
• sign in by an encrypted email address to exercise the blind index,
• complete an existing MFA login to exercise the secrets master key.

Use probes that exist in your app. All three must succeed.

Check audit activity

Open the Scaleway audit log. The rollout should produce a small burst of decrypt calls that matches the number of new pods and slots. Repeated decrypt calls from a stable pod set need investigation.

Roll back

Use rollback when a pod cannot start, a round-trip probe fails, Key Manager is unavailable, or decrypt latency is unacceptable.

1. Stop the rollout. Do not migrate another app.
2. Restore all three plaintext variables from the password manager.
3. Deploy the infrastructure revision that selects the plaintext branch.
4. Wait for rollout success.
5. Confirm source=plaintext-env in the boot log.
6. Repeat the old-data, login, and MFA probes.

Do not use pulumi config rm as the rollback if the current infrastructure code uses config.require(...) for ciphertext. That fails before Pulumi can deploy. The reliable recovery path is the previous infrastructure revision or an already-reviewed plaintext branch.

After recovery, revoke a suspected decrypt token and investigate before trying the migration again.

Key Manager outages

The resolver gives each decrypt request five seconds. It retries server errors and timeouts twice, after 200 ms and 800 ms. Client errors such as 401, 403, and 404 fail immediately.

Running pods continue because they keep plaintext keys in memory. New pods stay unready. Avoid voluntary restarts during an outage. If capacity or node failure requires new pods, use the tested plaintext recovery procedure.

See also

• Rotate encryption keys
• Crypto-shredding
• Deploy to K3s