Routing on token price optimises the wrong number. What you actually pay for is completed work — and three retries on a cheap model beats nothing.
A task costs $0.02 on the small model and $0.20 on the large one. Obvious choice.
Except the small model gets it wrong twice before getting it right, each attempt burns a review cycle, and the third attempt is subtly wrong in a way nobody catches for a week. The $0.02 task cost $0.06 in tokens and considerably more in everything else.
Most LLM routers optimise for cost per token. That’s measurable, which is why it’s popular, and it’s the wrong denominator. What you care about is cost per unit of work completed — where the unit is a task done correctly the first time.
Those two numbers diverge exactly where it matters: on hard tasks, where retries cluster.
AgentStateRouter — the Agent Execution Optimization Engine — routes on what actually happened in your workload:
The history is the point. A generic benchmark tells you how models rank on someone else’s tasks. Your routing table should be built from your own.
AEOE ships in three shapes over one engine: a suggester over HTTP/JSON-RPC that recommends without sitting in the request path, a proxy facade that’s OpenAI- and Anthropic-compatible for drop-in use, and an embedded crate for Rust applications.
Every routing decision is a commit on AgentStateGraph, with intent, reasoning, confidence and outcome. So “why did this task go to the expensive model” has an answer, and the answer improves the next decision.
Router is at v0.1.0-alpha, with the foundation in active development.
Next: we can’t reason about agents without agreeing what the words mean.