Provenance and cost control for AI pipeline steps

Enterprise Uses ai-pipeline, ai-foundation and prompt-store — part of the commercial Kumiko Enterprise offering.

What you get

ai-pipeline gives you three Tier-3 step kinds — ai.generate, ai.extract, ai.classify — that only run inside defineWorkflow. Each step resolves its own policy, provider and prompt at run time and writes an immutable provenance record once it’s done. Nobody has to trust “the pipeline probably used prompt v3, on whatever model was configured” — there’s a row that says so.

resolveStepPolicy merges a tenant’s stored overrides (enabled, providerId, model, step params) over the author’s code defaults. A TenantAdmin retunes a step live via ai-pipeline:write:set-policy — no deploy.
resolveProviderForStep resolves the actual LLMProvider for that policy (which vendor SDK, in the end, doesn’t leak past this line).
resolveActivePromptWithRevision pulls the active prompt text and its revision number from prompt-store — falling back to a code-supplied default if the store has no revision for that key yet.

export type AiStepProvenance = {
  readonly promptKey: string;
  // Absent when the store held no revision yet (fallback was used).
  readonly promptRevision?: number;
  readonly providerId: string;
  // Absent unless the effective policy pinned a model — the provider
  // interface exposes no resolved-model getter, so an unpinned step's
  // actual model isn't observable here. Foundation limitation, not an
  // oversight: don't read this field expecting it to always be there.
  readonly model?: string;
  readonly usage: TokenUsage;
};

Provenance never carries the resolved prompt text or the request/response content, only identifiers and token accounting — which is also the DSGVO-friendly shape: you can audit months of “which prompt touched this tenant” without ever loading anything a user actually typed or an LLM actually said.

A conflict Anthropic 400s on, and why the fix has to be automatic

Adaptive thinking and forced tool-use don’t mix, and Anthropic will tell you so at request time, not at review time:

Thinking may not be enabled when tool_choice forces tool use.

The naive path sets both because both sound like “make the model try harder” — thinking for better reasoning, forced tool-use for a guaranteed structured response. They’re incompatible on Anthropic’s side: forcing a tool call removes the model’s ability to “decide” anything, which is exactly what adaptive thinking needs room to do. The provider detects a forced tool_choice and disables adaptive thinking automatically before the request goes out, rather than leaving it to every caller to remember. Get this wrong once and every eval run using toolChoice: { type: "any" } against an Opus override breaks the same way, all at once.

The 25% that came from tightening, not switching models

The default model is Sonnet, not Opus — and it reaches pass-parity with Opus’s prior output at roughly a quarter of the cost, in an internal eval, once tool_choice and a tightened output schema were in place. The cost difference was never really “smarter model.” It was “fewer degrees of freedom to hedge before committing to an answer.”

A second, separate lever: effort: "high" instead of Anthropic’s own recommended "xhigh" for coding/agentic tasks. xhigh produced roughly twice the thinking tokens for the same pass rate in the same eval — thinking tokens are billed the same as output tokens, so that’s a straight 2x on a chunk of the bill for zero accuracy gain. This one only matters when Opus is used via an explicit override; Sonnet doesn’t get adaptive thinking at all.

A third, independent lever: the output token cap defaults to 4000, not 16000. For structured tool-call responses this is comfortably above what any real step needs — but capping it lower matters because Anthropic bills against max_tokens as an upper bound in some failure paths, not only against what the model actually emits. None of these three levers touch the prompt content or the provider. All three came from looking at per-step token usage, which only exists because of the provenance record above it.

Cache placement, and the 16k wall

Values above roughly 16k output tokens need streaming, or requests start hitting the SDK’s HTTP timeout before the response streams back far enough to matter. Prompt caching only pays for itself if the cache_control breakpoint sits on the last static block:

const system = cacheableSystemMessages.map((message, i) => ({
  type: "text" as const,
  text: message.content,
  // Only the LAST system block gets the breakpoint — this caches
  // `tools + system` together as one prefix, which is what we want:
  // stable tool definitions and instructions, cached once, reused
  // across calls with different dynamic messages.
  ...(i === cacheableSystemMessages.length - 1
    ? { cache_control: { type: "ephemeral" as const } }
    : {}),
}));

Put the breakpoint on a block that changes per request (accidentally include something per-tenant above it) and you get a silent cache-miss that looks identical to a hit in the logs, while you keep paying full price on every call.

Why this is event sourcing, structurally

A step’s provenance record is never updated. A retry, a backfill, a manual re-trigger — each gets a new record, not a patch to the old one. The pipeline’s current behavior is a projection over “what actually ran, ever,” the same way any event-sourced read model is a projection over an append-only log. The payoff is the same too: “what changed” and “why did this tenant’s output differ from that one” are answerable by reading history, not by trusting that today’s code matches what ran three weeks ago.