Skip to content
Kumiko

Crypto-shredding & blind index

In plain words

Section titled “In plain words”

Imagine every person’s data sits in a locked box, and every person has their own key. As long as the key exists, the app can open the box and show or send that person’s data. When someone says “delete me”, we do not hunt down every copy of their data — we destroy their key. From that moment, every copy — in the database, in the history log, even in yesterday’s backup — is unreadable scrap, forever. Nobody can undo it, not even us. That is the whole idea; the rest of this page explains how Kumiko does it and where the sharp edges are.

one person = one key ──unlocks──▶ their data (rows, history, backups)
"delete me"          ──destroys the key──▶ all of it unreadable, at once

Why it exists

Section titled “Why it exists”

Kumiko is event-sourced: events are immutable, and personal data written into them cannot be rewritten later. Crypto-shredding resolves that tension. Every subject (a user or a tenant) gets its own data-encryption key (DEK). Annotated fields are encrypted under the subject’s key before they reach the row or the event log. “Forget this person” (GDPR Art. 17) then means: erase the key. Every copy of the data — projection rows, the event log, yesterday’s backups — becomes permanently unreadable in one operation. No data hunt, no event rewriting.

What it does

Section titled “What it does”
  • Encrypts at the only write path. Fields annotated pii, userOwned or tenantOwned are encrypted inside the event-store executor, so live writes and projection rebuilds behave identically. The stored value is a self-describing ciphertext (kumiko-pii:v1:<subject>:<blob>), which names its own subject — no lookup tables needed for cleanup.
  • Decrypts on framework read paths. Executor reads (detail, list, write-responses) return plaintext. After the key is erased they render the [[erased]] sentinel instead.
  • Keeps equality lookups working via the blind index (below) — login by email, signup duplicate checks, password reset and invite flows work on the encrypted column.
  • Erasure is immediate and total. kms.eraseKey(subject) — shipped as an operator command by the crypto-shredding bundled feature, and executed automatically by user-data-rights after the deletion grace period. The forget pipeline also nulls the subject’s blind indexes right away, closing the linkage window.

Annotate your fields

Section titled “Annotate your fields”

Three orthogonal subject annotations decide whose key protects a value: pii: true (the row itself is the subject), userOwned: { ownerField } (the value is about the user named in another field), and tenantOwned: true (the current tenant). The tested mini-HR recipe shows all the moving parts:

import {
  createEntity,
  createTextField,
  defineFeature,
  registerEntityCrud,
} from "@cosmicdrift/kumiko-framework/engine";


export const employeeEntity = createEntity({
  table: "read_hr_employees",
  fields: {
    displayName: createTextField({ required: true, pii: true }),
    email: createTextField({ required: true, format: "email", pii: true, lookupable: true }),
    department: createTextField({ sortable: true }),
  },
  softDelete: true,
});


export const hrCommentEntity = createEntity({
  table: "read_hr_comments",
  fields: {
    employeeId: createTextField({ required: true }),
    body: createTextField({ required: true, userOwned: { ownerField: "employeeId" } }),
    authorName: createTextField(),
  },
  softDelete: true,
});


const hrWrite = { access: { roles: ["Admin"] } } as const;
const hrRead = { access: { roles: ["Admin"] } } as const;


export const hrFeature = defineFeature("hr", (r) => {
  registerEntityCrud(r, "employee", employeeEntity, {
    write: hrWrite,
    read: hrRead,
    verbs: { update: false, delete: false, restore: false },
  });


  registerEntityCrud(r, "hr-comment", hrCommentEntity, {
    write: hrWrite,
    read: hrRead,
    verbs: { update: false, delete: false, restore: false, list: false },
  });
});

📄 On GitHub: samples/recipes/crypto-shredding-hr/src/feature.ts

Boot wiring

Section titled “Boot wiring”

Crypto-shredding needs two secrets, deliberately kept apart:

// production boot — both keys come from the environment
runProdApp({
  features,
  // per-subject DEK store (encrypted at rest under the platform KEK)
  kms: createPgKmsAdapter({
    databaseUrl: env.SUBJECT_KEYS_DATABASE_URL,
    masterKey: env.PLATFORM_KEK,
  }),
  // separate 32-byte key for the HMAC blind index (openssl rand -base64 32)
  blindIndexKey: env.KUMIKO_BLIND_INDEX_KEY,
});

Boot fails loudly when the wiring is inconsistent — a KMS with lookupable fields but no blindIndexKey aborts instead of silently breaking every login lookup.

Equality lookups: the blind index

Section titled “Equality lookups: the blind index”

Encrypted columns cannot be queried — WHERE email = '[email protected]' never matches a ciphertext. Marking a text field lookupable: true makes the framework maintain a second, generated column (email_bidx) containing a keyed HMAC of the plaintext. Query compilers rewrite equality filters to (email = $1 OR email_bidx = $2), so the same query matches plaintext legacy rows and encrypted rows — the rollout needs no feature flag and no backfill precondition.

Properties to know:

  • Byte-exact. No case folding, no trimming. [email protected] and [email protected] are different values — normalize before you write, exactly as you already had to for a unique index.
  • Equality only. The index answers “is this exactly X?” — nothing else.
  • Erasure-aware. Erasing the subject key nulls the blind index (the forget pipeline immediately, rebuilds by construction), so a forgotten email stops matching.
  • Separate key. The HMAC key is not the platform KEK and is never seen by KMS adapters — rotating one does not touch the other.

Reading and writing outside the executor

Section titled “Reading and writing outside the executor”

The executor is the only path that encrypts and decrypts automatically. Everything else follows two rules:

// raw reads (fetchOne / selectMany) return the STORED value — ciphertext.
// Decrypt before comparing, re-querying or handing it to a mail/response:
const email = await decryptStoredPii(row.email, "my-feature:context");


// unmanaged direct-write stores (r.unmanagedTable) bypass the executor —
// encrypt annotated fields yourself and declare it at the registration:
await insertOne(db, myTable, await encryptForDirectWrite(myEntity, row, "my-feature:create"));
r.unmanagedTable(buildEntityTableMeta("my-entity", myEntity), {
  reason: "read_side.direct_write",
  piiEncryptedOnWrite: true, // boot fails without this once PII fields exist
});

Built-in guardrails

Section titled “Built-in guardrails”

Forgetting a decrypt should fail a test, not leak in production. The framework enforces this in layers:

  1. Boot gates. lookupable without a subject annotation, searchable/ sortable on an encrypted field, PII fields on an unmanaged store without piiEncryptedOnWrite, a KMS without a blind-index key — all abort boot with an actionable message.
  2. GDPR export decrypts centrally. Art. 15/20 bundles carry the data, never the blob: every kumiko-pii: value in any export hook’s output is decrypted in one shared pass (or exported as [encrypted:unavailable] when no KMS is configured).
  3. Runtime tripwires. A ciphertext in a JSON API response turns into a loud 500 (pii_ciphertext_leak) in dev/test and is redacted + logged in production. Outgoing mail to a ciphertext recipient is always refused; ciphertext in a mail body throws in dev and is redacted in production.

What it does NOT do

Section titled “What it does NOT do”
  • No substring search, no sorting, no LIKE on encrypted fields — the ciphertext has no usable order. searchable/sortable on a subject- annotated field is a boot error, by design.
  • No normalization. The blind index matches byte-exact; it will not make a case-insensitive login out of a case-sensitive one.
  • No secrecy against equality probing. Anyone who can issue queries and already knows a candidate value can test “is it exactly this?” until the key is erased. The blind index is deterministic on purpose — it is a lookup mechanism, not an additional secrecy layer.
  • No retroactive protection. Values written before a KMS was configured keep their plaintext bytes in old events and backups until the backfill lands (kumiko-framework#799).
  • No row deletion. Erasure makes values unreadable; rows, events and audit trails stay. Combine with data-retention policies when rows themselves must go.
  • No coverage for app-provided senders. The mail tripwire guards every transport built through createTransportForTenant and the delivery email channel. A custom callback that ships addresses to an external API on its own is outside the framework’s reach — decrypt first (decryptStoredPii).
  • No remote/BYOK crypto yet. The KMS contract is local-key (envelope encryption in-process); Vault-transit style adapters are future work.
Section titled “Related”