← All products Testing and validation

AgentStateCrucible

One passing run is a sample, not a test.

Early access
// The problem

Agents are non-deterministic, so a single successful run proves almost nothing. Exact-match assertions are too brittle for generated text, and pass/fail hides the thing that actually regressed.

// The premise

Run the same scenario against multiple agents from the same starting state, capture every decision as a commit, and score the runs comparatively with a judge agent. Two agents can reach the same answer by very different routes; only trajectory comparison sees the difference.

// Capabilities
  • Same scenario, same starting state, same policy for every candidate agent
  • Every decision a commit — intent, reasoning, confidence, alternatives, authority
  • A pluggable LLM judge scoring runs side by side
  • Sealed, Merkle-rooted epochs an auditor can verify without trusting the harness
  • Measures correctness, reasoning quality, effect blast radius and cost

Scenario runner · judge scoring · run diffing · sealed audit epochs

// Pairs with
  • AgentStateDeveloper — validate behaviour and blast radius together
// Licensing

Source-available under BSL-1.1, converting to Apache 2.0 18 months after each version's first publication. Internal use — including in production — is unrestricted.

// The rest of the suite

Built on the same graph.