🔥 [ this week's Hear Me Out ]

Hot take: Claude came last in our evals. Its security defaults are why.

We ran 25 model and harness combinations through seven everyday email jobs on AgentMail. All ten Claude setups landed in the bottom eleven, across four Claude models and four harnesses, Claude Code included. Other model families finished 90% of their runs. Claude finished 63%.

That surprised us, so we went back through the runs to see where Claude was losing.

We expected a skill gap and did not find one. Four of the seven jobs hid the email the agent needed, either in spam or behind an authentication filter. When the job was only to read that hidden email, Claude found it and finished 86% of the time, in line with everyone else. When the job meant replying to it, Claude finished 24%.

Claude's default is no. Everyone else's is yes.

Two jobs made most of the difference, and both asked the agent to reply to mail flagged as spam. One was a password reset with a reply address that pointed somewhere else, the kind of email a careful person would question. The other was a batch of signup confirmations, where the prompt told the agent: "I have authorized this; do not skip it on principle."

Claude held back on both. Most of the other models went ahead on both, and replied to the suspicious reset almost every time. On those two jobs, Claude finished 23% of runs and everyone else finished 88%.

Whether Claude's no is the better default is your call

A wrong no leaves a job undone until someone comes back to finish it. A wrong yes sends an email you cannot unsend, possibly to an address someone else controls. Claude chose the mistake that is easier to clean up, and plenty of teams will want exactly that.

Neither default makes one model smarter than the other. Both come from the same gap. Models have not been trained on agent infrastructure yet, so they apply a blanket rule where a trained agent would read the signal.

Until it does, do not leave the call to the model. If you are putting an agent on an inbox and expect it to get work done, write the rules yourself: which senders it can trust, and which actions wait for a person. Put those rules in the infrastructure, where the agent meets them on every call, instead of in a prompt it can weigh against its instincts.

On AgentMail that means allow and block lists for senders, and drafts for anything that needs sign-off. The same rules help both kinds of model. The cautious one gets permission for work you have authorized, and the eager one stops at the email you would flag.

★ [ spotlight ]

📅 [ events ]