A better model can make your agent worse
A stronger model with higher scores looks like a free upgrade, until the agent that worked last week starts getting things wrong, quietly. Here is what happened when we ran one agent on two frontier models and changed nothing else.
Upfront information dump
A customer opens your support agent with this:
The multi-turn failures that prompt evals can't see
Most agent failures we see in pilots don't show up on prompt evals.
The failed Tool Call when Simulating a Customer Conversation Across Three LLMs
Recently, to assess AI Agent performance with tool calls, we executed the same multi-turn conversation across the three tiers of OpenAI's GPT-5.4: standard, mini, and nano.
The Risks of Agent Speculation
It's no surprise that hallucinations are a common known failure during agentic AI testing. The agent starts to overpromise, begins to fabricate answers and even claims that it…