The test standard
Kumiko apps and the framework itself share one test setup, delivered by
@cosmicdrift/kumiko-testing and scaffolded by kumiko new app. This guide
explains what the standard is and why it looks the way it does, so the old
patterns — workers: 1 against shared state, a raised timeout, a real test
that fires because a key happens to be in .env, a hand-copied screenshot
runner — stop coming back out of habit. For diagnosing a red or flaky test,
see test-failures.md; this guide is the model behind
those steps.
Four classes, never mixed
Section titled “Four classes, never mixed”| Class | Files | Command | Budget |
|---|---|---|---|
| Unit | *.test.ts | bun run test | 5 s per test |
| DOM | *.test.tsx | bun run test:dom | 5 s per test |
| Integration | *.integration.test.ts | bun run test:integration (files run in parallel) | 15 s per test |
| Build | *.build.integration.test.ts | bun run test:build (bun kumiko-testing integration --build), own CI job | 180 s per test |
| E2E | e2e/*.spec.ts | bun run e2e | fixed in defineAppE2eConfig |
| Real provider | *.real.test.ts, *.real.spec.ts | bun run test:real, bun run e2e:real | 240 s per test |
Tests that run bun install, a build or a Docker image in a subprocess are build tests. test:integration skips them; the app runs them in its own app-<name> CI job, which only runs when the affected feature changes or nightly.
Unit tests carry no services and stay fast enough to run on every save.
Integration tests hit a real Postgres/Redis stack and prove the app’s own
wiring. E2E tests boot the app and prove the browser contract. Mixing them —
a unit test that reaches for Postgres, an integration test that spins up a
browser — loses the fast/clear-signal property of the cheaper class and
slows down the whole suite for no extra coverage. test:dom (*.test.tsx,
happy-dom via preload/dom) sits next to unit tests for component-level
rendering that needs a DOM but not a browser.
kumiko-testing bunfig generates the bunfig files (preloads, path ignores)
that keep each class in its own lane. Bun ignores a [test] timeout key, so
the budget above is a --timeout flag in the scripts, from one constant
(TEST_TIMEOUT_MS) — not something an app sets per file.
Unit, DOM, integration and real all preload SCHEMA_ENV_DEFAULTS
(JWT_SECRET, KUMIKO_SECRETS_MASTER_KEY_V1,
KUMIKO_SECRETS_MASTER_KEY_CURRENT_VERSION) — the env keys the framework’s
own schema (auth, secrets, field-encryption) requires, each filled with ??=
so an app’s own value always wins. Integration and real additionally preload
SERVICE_ENV_DEFAULTS (DATABASE_URL, REDIS_URL, …); unit and DOM don’t,
so a unit test that accidentally reaches for a service still fails instead of
silently connecting. An app-specific default (a demo email, a seed value) is
never added to either map — it stays in the app’s own preload extra, kept
across regenerations by kumiko-testing bunfig.
Real providers: opt-in, never by accident
Section titled “Real providers: opt-in, never by accident”Tests that call a real external provider (LLM, mail, payment) are named
*.real.test.ts (Bun) or *.real.spec.ts (Playwright) and start with
requireRealProviders() from @cosmicdrift/kumiko-testing/helpers (Bun)
or @cosmicdrift/kumiko-testing/e2e (Playwright specs and configs, which load
under Node and must not import the ./helpers barrel). Three
reasons they are walled off from the default run:
- Cost. A real LLM or payment call has a price; the default suite runs on every save and every PR, a real-provider test should not.
- Non-determinism. An external provider’s output isn’t fully under the test’s control, so it can go red for reasons that have nothing to do with the change under test.
- Secret leaks. Talking to a real provider means a real credential in the process — the default CI run never holds one.
They run only when KUMIKO_REAL_PROVIDERS=1 is set, through the test:real /
e2e:real scripts, and never in CI: requireRealProviders() throws the
moment CI is set, regardless of the flag. A provider API key sitting in the
environment does not enable them on its own — the flag is a separate,
explicit switch.
To run one on purpose: bun run e2e:real -- <file> (or test:real for Bun
tests). To add one, write the shared flow once and wrap it in a thin
*.real.spec.ts that calls requireRealProviders(). Neither the wrapper nor
the shared flow sets a timeout: both scripts take the real-provider budget from
one constant, E2E_TIMEOUT_MS.real (test:real passes it as --timeout via
TEST_TIMEOUT_MS.real, defineAppE2eConfig applies it only when
KUMIKO_REAL_PROVIDERS=1 is set). It is 240 s because solon’s
document-onboarding flow, the slowest real-provider run today, needs that
long; the default suite keeps its own budget.
Parallel by default
Section titled “Parallel by default”Every flow seeds its own tenant with seedTenant() — in-process for
integration tests, over the seed routes the app’s e2e/server.ts mounts with
createE2eSeedRoutes() for e2e (those routes 404 unless
KUMIKO_TEST_SEED=1, NODE_ENV is not production, and the per-run token
matches). A tenant per flow is what makes running everything in parallel
safe: two flows writing to the same tenant, user or row race each other,
workers: 1 only hides that race by never letting it happen, and the race
comes back the moment two people run the suite differently or a CI runner
gets a second core. Fix the shared state, don’t serialize around it.
Seeded data that has to look old (a 90-day uptime strip, a chart with history) is written with runSeedWritesAt(createdAt, fn) from @cosmicdrift/kumiko-framework/event-store. Every event appended inside fn goes through the normal write path (handlers, validation, projections) but is stored with createdAt instead of the database now(). The function throws SeedModeDisabledError before it runs fn unless KUMIKO_TEST_SEED=1 and NODE_ENV is not production, and the server refuses to boot when the flag is set together with NODE_ENV=production. The back-dated time lives in an AsyncLocalStorage, so nothing parsed from a request can set it; only in-process code such as an extraSeeders entry of createE2eSeedRoutes can call it.
const seedBackdatedNote: E2eExtraSeeder = async (ctx) => { const createdAt = Temporal.Now.instant().subtract({ hours: 90 * 24 }); return runSeedWritesAt(createdAt, async () => ctx.dispatch.write("notes:write:note:create", { title: "old" }));};A global-admin view is the one place a tenant per flow doesn’t isolate you.
A SystemAdmin overview (show-pony’s platform-overview, a tenant list, a
platform-wide counter) sees every tenant, including the ones parallel flows are
seeding at that moment. Seed the operator per flow with
tenant.addUser(["SystemAdmin"]) instead of sharing the tenant seeded at boot,
then assert only on the tenants your own flow seeded (find them by their name
or id), never on global totals, counts or “the first row”: those change with
whatever else runs next to the flow. SystemAdmin is seedable because the seed
gate above is the boundary around every seed route; nothing else guards it.
Test data a flow can’t create through the UI or the seeded users comes from an
app seeder: createE2eSeedRoutes({ extraSeeders }) on the server,
tenant.seed(name, body) in the flow. It sits behind the same gate, only
reaches tenants that server seeded, and writes through the app’s own handlers
as the tenant’s system user (SystemAdmin), never into tables directly. The
target handler must admit SystemAdmin; data no normal handler produces
(backdated history, for example) needs a SystemAdmin-only handler for the
seeder to call.
defineAppE2eConfig sizes workers off the CPU count rather than maximizing
them: on a small (1.5-CPU) CI runner, 1/2/4 workers measured 2.6/2.6/2.7 min,
so the run is already saturated at one worker and more only adds Chromium
processes. A project that sets its own
workers throws.
Specs that change app-wide state
Section titled “Specs that change app-wide state”All specs of an app run in parallel against one server. A spec that changes
state every tenant sees, such as a catalog channel or a global setting, can
break specs running next to it, while both pass on their own. Mark such specs
with isolatedSpecs instead of lowering the workers:
export default defineAppE2eConfig({ port: 4321, isolatedSpecs: ["**/composer.spec.ts"],});The marked files leave the chromium project and every app project and move
into an isolated twin of it (chromium-isolated, <name>-isolated) that keeps
the project’s own use. The twins start once all other projects have
finished and run their files one after another on a single worker. A project
with its own testMatch is rejected. Without the
option the config is unchanged.
The isolated project depends on the parallel projects, so Playwright’s rules for dependencies apply:
- If a spec in a parallel project fails, the isolated project does not run. Fix the parallel failure first.
- A file or
-gfilter that hits an isolated spec runs the full parallel suite first. Add--no-depsto run only the isolated spec. - Playwright does not shard dependency projects, so one shard would get every
spec.
defineAppE2eConfigthrows whenisolatedSpecsmeets--shard. - Every app project needs a
name, because the isolated project waits on it by name.
Concurrency
Section titled “Concurrency”Tests in one file run one after another, so two handler calls only overlap when a single test fires them together. Promise.all over stack.http.write is the whole mechanism. Two helpers in @cosmicdrift/kumiko-testing/helpers make the recurring shapes deterministic.
burst fires N requests at once and tallies the statuses. It throws before sending anything when N exceeds stack.dbPoolMax: every request inside a transaction holds a pool connection, so a larger burst queues on the pool instead of running concurrently, and a handler that waits for a peer deadlocks. Raise dbPoolMax in setupTestStack or lower N.
const { statuses } = await burst(stack, 6, () => stack.http.write(QN, {}, user));expect(statuses[200]).toBe(3);expect(statuses[429]).toBe(3);createBarrier proves that calls were inside the handler at the same time. The handler under test calls arrive() and continues once expected calls have arrived. When a request is refused before it reaches the handler (a cap gate, for example), the refusing path calls skip() so the arrived calls are not left waiting. The barrier rejects after timeoutMs (default 5000) with the arrival count, so serialized calls fail loudly instead of passing by accident.
const barrier = createBarrier({ expected: 2 });// inside the handler under test:await barrier.arrive();// in the test:const { statuses } = await burst(stack, 2, () => stack.http.write(QN, {}, user));expect(statuses[200]).toBe(2);expect(barrier.arrived).toBe(2);A sleep is no substitute for the barrier. It passes when the calls happen to overlap and fails when the machine is slow, which is exactly the flakiness this document rules out.
Environment and streams
Section titled “Environment and streams”Use withEnv instead of bare process.env assignments. With --parallel=4 --no-isolate a variable that is not reset leaks into other files running in the same worker. withEnv restores the previous values in a finally, and a key that was unset before is deleted again rather than set to the string "undefined".
await withEnv({ FEATURE_X: "on", LEGACY_KEY: undefined }, async () => { await runScenario();});Use readSseFrames instead of a hand-written stream parser. It buffers across chunk boundaries, joins multi-line data: fields, skips pings, and stops at the first frame that satisfies until. It throws when the stream ends early or the timeout (default 5000 ms) passes, and always cancels the reader.
const frames = await readSseFrames(response, { until: (frame) => frame.event === "entity.changed",});expect(JSON.parse(frames.at(-1)?.data ?? "{}")).toMatchObject({ id });Apps with MFA
Section titled “Apps with MFA”Against an app that enforces MFA, the e2e seedTenant fixture enrolls the
seeded admin itself: when the first login answers with a required MFA setup,
it enrolls a confirmed TOTP factor through the real auth-mfa endpoints.
seedTenant({ mfa: "totp" }) forces the enrollment when no policy demands it.
The base32 secret comes back as admin.mfaTotpSecret, and loginAs,
loginViaApi and loginViaUi answer the MFA challenge with it.
tenant.addUser(roles, { mfa: "totp" }) enrolls another user the same way and
returns the secret on that user. When an account needs MFA and the credentials
carry no secret, loginViaApi throws. This needs @cosmicdrift/kumiko-testing
0.345 or later.
Parallel runs
Section titled “Parallel runs”Framework integration runs as one bun test --parallel=4 --no-isolate call
over every *.integration.test.ts file, built by the same
buildIntegrationTestArgs the CLI uses. The framework’s unit step uses the
same flags; DOM steps stay serial. Perf gates run serially.
--no-isolate is required: bun’s --parallel implies --isolate, which
leaks native memory per test file and can OOM-kill a small CI runner. It is
safe because the template isolates through data (seedTenant per flow, a
queue prefix per stack), not through OS processes. The guard-integration-mocks
check (part of kumiko check) also ratchets mock.module, the one way files could leak state into each other.
The CI Postgres runs with fsync=off, synchronous_commit=off and
full_page_writes=off: DROP DATABASE in stack.cleanup() forces a
synchronous checkpoint that queues up under concurrent teardowns.
Result for the framework integration suite: about 460 s serial against 170 to
215 s with --parallel=4.
Under --parallel, bun’s workers ignore coveragePathIgnorePatterns and
report more executable lines with zero hits, so unit line coverage reads
lower without any test covering less. scripts/coverage-badge.ts filters
all lcov inputs against bunfig.ci.toml itself.
Boot validation in test stacks
Section titled “Boot validation in test stacks”setupTestStack checks nav, workspace and tree-action references of the mounted
features, the same checks the prod boot runs. A reference into a feature that is
not mounted is skipped, because tests usually mount a subset; a reference into a
mounted feature that does not resolve throws. Screen and ref-entity checks span
features and run only with setupTestStack({ validateBoot: "full" }). Every app
should have one boot test that mounts the prod feature composition with
validateBoot: "full", so a broken screen fails in CI instead of at deploy. validateBoot returns the boot warnings it logs ({ warnings }, each with kind, optional feature and message); a boot test asserts on that result instead of spying on the console.
Timeouts and retries belong to the template
Section titled “Timeouts and retries belong to the template”defineAppE2eConfig owns timeouts, retries (always 0) and workers; no
per-test test.setTimeout, no per-project override. Centralizing them is
what makes the budget in the table above mean something — a timeout an app
can silently raise is not a budget, it’s a suggestion. When a test is
actually slow or flaky, the fix is almost never the timeout: see
test-failures.md for the diagnosis
order and the // kumiko-lint-ignore test-timeouts #<issue> <reason> marker for the rare
case that needs one. Every guard opt-out uses this one marker syntax; for a whole file the
form // kumiko-lint-ignore-file <slug> <reason> goes in the leading comment before the first statement.
Screenshots: one runner, not a local copy
Section titled “Screenshots: one runner, not a local copy”Doc and preview screenshots go through runScreenshots/runMatrix
(@cosmicdrift/kumiko-testing/e2e), not a hand-written page.screenshot
call. SCREENSHOT_DIR is mandatory — screenshot specs are skipped entirely
unless it’s set, and the runner writes to $SCREENSHOT_DIR/<name>.png
(runScreenshots) or $SCREENSHOT_DIR/<name>/<locale>/<theme>/<viewport>.png
(runMatrix) so the images that end up committed into docs are always
produced the same way, from the same viewport and device-scale-factor
constants, never from whatever an app’s own config happened to set. runMatrix’s desktop
viewport is 1920 wide; set SCREENSHOT_DESKTOP_WIDTH=1440 for a design review at
another width.
There is no settleMs. A fixed sleep before the capture is exactly the
“wait for time, not a condition” anti-pattern the diagnosis order warns
about — it’s either too short (captures mid-animation) or too long (slows
every run for a margin nobody measured). The runner instead polls until no
data request is in flight and the page’s DOM/scroll-size/animation
fingerprint has been stable for a few consecutive polls, then captures.
Template instead of copy
Section titled “Template instead of copy”@cosmicdrift/kumiko-testing is an API — defineAppE2eConfig,
createE2eSeedRoutes, seedTenant, runScreenshots/runMatrix — not a file
an app copies and edits. A copied playwright.config.ts drifts the moment
the template gains a fix (a new timeout constant, a worker-count change,
a screenshot-stability fix): the copy doesn’t get it until someone remembers
to re-copy it, and usually nobody does. Calling the API instead means an
improvement ships to every app, including third-party ones, through an
ordinary version bump of @cosmicdrift/kumiko-testing — no coordinated
find-and-replace across repos. The test-template-drift guard enforces this:
a config-level viewport or a deviceScaleFactor anywhere is flagged, and a
// kumiko-lint-ignore test-template-drift #<issue> <reason> marker directly above it
suppresses the finding.
A spec’s own test.use({ viewport }) is not flagged — the template defaults
to a 1920px desktop viewport, and a spec asserting a layout that only exists
below that width (a mobile breakpoint, a narrow-container overflow case) sets
its own viewport, same as it always could. deviceScaleFactor has no such
per-spec case, so it stays template-owned everywhere.
Coming from the old setup
Section titled “Coming from the old setup”| Old pattern | New pattern |
|---|---|
workers: 1 / fullyParallel: false in playwright.config.ts | defineAppE2eConfig({ port, serverEntry, ... }) — parallel by default, tenant per flow |
Hand-rolled defineConfig with a custom viewport/DPR | defineAppE2eConfig (viewport/DPR are template-owned) |
test.setTimeout(n) / a raised project timeout | Diagnose first (test-failures.md); kumiko-lint-ignore test-timeouts marker only for a genuine no-condition wait |
| A real-provider test that runs whenever an API key is present | *.real.test.ts / *.real.spec.ts + requireRealProviders(), gated by KUMIKO_REAL_PROVIDERS=1, hard-stopped in CI |
A local page.screenshot() helper for docs images | runScreenshots / runMatrix with SCREENSHOT_DIR set |
| A shared/global test user across flows | seedTenant() per flow |
A copied playwright.config.ts / bunfig from another app | kumiko-testing bunfig generator + defineAppE2eConfig, updated by a version bump |
A local test-setup/dom.preload.ts copy | kumiko-testing bunfig --dom, which preloads @cosmicdrift/kumiko-testing/preload/dom |
An app-local preload setting JWT_SECRET / KUMIKO_SECRETS_MASTER_KEY_* | SCHEMA_ENV_DEFAULTS, preloaded automatically in every kumiko-testing bunfig variant |