We’re open-sourcing parallel-search-evals, the harness we use to measure Parallel Search, test new models, and power our Search Capability Leaderboard. | | Explore the Search Capability Leaderboard to see some results from models tested with the harness. | What's inside | Any model. Run it through OpenRouter, OpenAI, Anthropic, Google, or any OpenAI-compatible endpoint. Native adapters keep each provider's own tool-calling and reasoning protocols intact. | Five modes. Each run tests one model on one benchmark in one mode: turbo, fast, basic, advanced, or baseline (no tools). Compare baseline to a search run to see the lift from search. | Two tools, fixed budget. The model gets Parallel's Search API and Extract API, with 20 tool calls per question. It's told to rely only on what it finds, not what it already knows. | Official grading. A GPT-6 Sol judge grades each answer with the benchmark's official prompt: F1 for DeepSearchQA (partial credit for incomplete answers) and strict correctness for HLE. | Built for long runs. Per-call cost tracking and recovery when a provider connection drops halfway through a long research trace. | Your own questions. Add a suite with your own questions, reference answers, and grading rules. | Try it | The repo includes an agent skill, so you can point Claude Code or Codex at it and ask it to run the eval. | git clone https://github.com
cd parallel-search-evals
uv sync --frozen | | |
|