Provenance and cost control for AI pipeline steps

ForFramework developerSecurity engineer

LevelAdvanced

Enterprise Uses ai-pipeline, ai-foundation and prompt-store, part of the commercial Kumiko Enterprise offering.

What you get

ai-pipeline provides three Tier-3 step kinds: ai.generate, ai.extract, and ai.classify. They run inside defineWorkflow. Each step resolves its policy, provider, and prompt at runtime, then records provenance for the run.

resolveStepPolicy merges a tenant’s stored overrides (enabled, providerId, model, step params) over the author’s code defaults. A TenantAdmin retunes a step live via ai-pipeline:write:set-policy, no deploy.
resolveProviderForStep resolves the provider for that policy.
resolveActivePromptWithRevision pulls the active prompt text and its revision number from prompt-store, falling back to a code-supplied default if the store has no revision for that key yet.

export type AiStepProvenance = {
  readonly promptKey: string;
  // Absent when the store held no revision yet (fallback was used).
  readonly promptRevision?: number;
  readonly providerId: string;
  // Absent unless the effective policy pinned a model, the provider
  // interface exposes no resolved-model getter, so an unpinned step's
  // actual model isn't observable here. Foundation limitation, not an
  // oversight: don't read this field expecting it to always be there.
  readonly model?: string;
  readonly usage: TokenUsage;
};

Provenance stores identifiers and token accounting. It does not store the resolved prompt, request content, or model response. The in-memory step result does contain the generated or extracted value, so do not persist or log that value unless its data classification permits it.

Forced tool use and adaptive thinking

Anthropic rejects adaptive thinking together with forced tool use:

Thinking may not be enabled when tool_choice forces tool use.

The provider disables adaptive thinking when toolChoice is { type: "any" } or { type: "tool", ... }. Its tests cover both combinations. auto does not force a tool call and keeps adaptive thinking enabled for Opus.

Cost tuning

The Anthropic provider defaults to Sonnet 4.6. The provider package records an internal evaluation in which tool choice and a tighter output schema reached the earlier Opus pass rate at roughly a quarter of the cost.

A second lever is effort: "high" instead of "xhigh" for Opus overrides. The provider notes report roughly twice the thinking-token use at xhigh for the same pass rate. Adaptive thinking is only used for Opus; the provider suppresses it for other models.

A third lever is the output token cap. The Anthropic provider’s default is 4000; steps can override it with the policy’s maxTokens parameter. A response that reaches the cap is marked truncated for ai.generate.

Cache placement

Keep large responses on the provider’s streaming path. Prompt caching works as intended when the cache_control breakpoint sits on the last static block:

const system = cacheableSystemMessages.map((message, i) => ({
  type: "text" as const,
  text: message.content,
  // Only the LAST system block gets the breakpoint, this caches
  // `tools + system` together as one prefix, which is what we want:
  // stable tool definitions and instructions, cached once, reused
  // across calls with different dynamic messages.
  ...(i === cacheableSystemMessages.length - 1
    ? { cache_control: { type: "ephemeral" as const } }
    : {}),
}));

If a per-request value appears before the breakpoint, the cache prefix changes and the request becomes a cache miss.

What the record contains

Each result branch records succeeded, failed, or skipped. A record can include the provider, responded model, prompt revision, token usage, and error message. skipped records have no provider or token usage because no provider call ran. The prompt and model response are not stored in the step provenance payload.