|
🔥 [ this week's Hear Me Out ]
Introducing AgentMail Evals. Are frontier models bad at tool use?
We evaluated 25 model/harness combinations. Here is what we found.
We have noticed a pattern in how agents use tools. When something is unfamiliar, they sometimes reach for the patterns they already know. They find an operation that sounds right and use it before discovering what else is available. Or they accept the first result and stop looking.
You can have a capable model, a working API, and a task that still does not get done in the most efficient possible way.
As we hand agents more work across more products, this gets expensive. A model needs to figure out how the product actually works, and its harness needs to give it a fair chance to do that.
We wanted to see where things stand. How well do leading models and harnesses handle a tool when we give them a job and leave them to figure out the details?
So we gave them seven email jobs
Using AgentMail as the proxy, we tested 25 model/harness configurations. GPT, Claude, and Grok were among the models, running in setups including Codex, Hermes, OpenClaw, and OpenCode.
The tasks included finding a confirmation code, saving an attachment, preparing replies, and replying to a thread with someone added on BCC.
We described the outcome we wanted without telling the agent which endpoint or parameter to use. Each configuration ran with and without links to the relevant docs.
Then we checked the recorded actions and outputs. If an agent said it had finished, we checked whether it actually had.
|
Top score
GPT-5.5 in Codex had the highest observed overall score:
96.8 / 100
GPT-5.5 · Codex
|
The harness matters as much as the model
Claude Sonnet 5 gave us a particularly interesting comparison. It scored 68.9 in Hermes and 38.1 in OpenClaw on the same task bank.
|
Claude Sonnet 5 · same model, two harnesses
A 30.8-point spread for the same model across two harnesses. It does not tell us the exact cause, but it shows why choosing a model and evaluating the setup you ship are separate decisions.
|
Sometimes, reading the docs made things worse
On the reply-all task, the agent also needed to add a manager on BCC.
Our reply-all endpoint sounds like the obvious choice. Except it could not add that recipient. The working route was the ordinary reply endpoint with reply-all enabled and the manager in BCC.
|
Reply-all task · success rate
Docs helped on six of the seven tasks. Here, they led agents toward an operation whose name fit better than its capabilities did. That gives us a concrete question to fix in our own docs.
|
Other agents stopped at an empty default inbox view, even though the requested mail existed outside it. That gives us a concrete question to investigate: did the response make clear what the agent had not searched?
We would love your take
The full report covers the tasks, methodology, and results across the tested configurations. These findings come from AgentMail workflows; the details matter when deciding what they mean for yours.
It would mean a lot to us if you read it. Find your model and harness, inspect the failures, and tell us what you think we should test next.
|
