A better model can make your agent worse
A stronger model with higher scores looks like a free upgrade, until the agent that worked last week starts getting things wrong, quietly. Here is what happened when we ran one agent on two frontier models and changed nothing else.
Upfront information dump
A customer opens your support agent with this:
The multi-turn failures that prompt evals can't see
Most agent failures we see in pilots don't show up on prompt evals.
The Risks of Agent Speculation
It's no surprise that hallucinations are a common known failure during agentic AI testing. The agent starts to overpromise, begins to fabricate answers and even claims that it…