/bench

DeepBench

Which AI model is the best agent driver? DeepBench gives every model the same two tools and the same real missions — multi-step jobs that chain live DeepAPI calls (scraping, email, research, web search) — and checks the outcome deterministically. One wrong step fails the mission.

run r202607091526 · Jul 9, 2026 · pass@3 · 10 models · 5 missions

#	Model	Score	Cost	Time
1	qwen3.7-maxAlibaba	5/5	$0.43	432s
2	gemini-3.1-pro-previewGoogle	5/5	$0.98	685s
3	claude-opus-4.8Anthropic	5/5	$1.57	403s
4	gpt-5.5OpenAI	4/5	$0.21	564s
5	gpt-5.4-nanoOpenAI	4/5	$0.36	398s
6	gemini-3.1-flash-liteGoogle	4/5	$0.67	2041s
7	deepseek-v4-proDeepSeek	3/5	$0.75	1020s
8	mistral-medium-3-5Mistral	3/5	$1.03	686s
9	claude-sonnet-5Anthropic	3/5	$1.98	862s
10	grok-4.3xAI	0/5	$0.32	897s

Missions

Compound missions, not toy prompts. Each chains 2–4 live API calls against the real web; grading is a deterministic check of the final answer or the produced email draft.

M1Front Page HuntScrape Hacker News live: find the current #1 story, then its comment page. Graded against the front page as it was during the attempt.2+ calls · final answer, live top story + comment count
M2Fresh Post OutreachScrape a company's newest LinkedIn post (hours old) and draft an outreach email containing its exact URL.2+ calls · email draft, post URL proven by the attempt's own scrape
M3The Right Alex WangDisambiguate a famous name with search + profile scrapes; impostor profiles exist, and the live headline is required.2+ calls · final answer, exact profile URL + live headline
M4Inbox-Driven TaskRead an instruction email from the inbox, research a version number that changes every few weeks, draft the reply.3+ calls · reply draft, code word + live version
M5Cross-Platform HuntHop GitHub → web search → X to pin down a person's handle and live follower count.3+ calls · final answer, handle + live followers within 10%

Method

One identical minimal tool-calling loop for every model: a DeepAPI request tool and a submit-answer tool. No model-specific prompts, no retries, no scaffolding — and no API cheat sheet: models must learn request shapes from the API's own error guidance.
Live ground truth: answers are graded against the world as it was during the attempt (front-page rankings, hours-old posts, current release versions, live follower counts), resolved independently of the model. Training-data memory cannot pass.
pass@3: a mission counts as passed if any of 3 attempts passes. Live web targets make single attempts noisy.
Hard limits per attempt: 14 model turns, 10 minutes, capped spend. Blowing a limit fails the attempt.
Cost is the full attempt cost: model tokens plus DeepAPI usage. Ranking: missions passed, then total cost, then total time.

DeepBench measures which models drive the DeepAPI agent API best — it is not a general model quality ranking. Not affiliated with Baidu's 2016 hardware benchmark of the same name or the 2025 code-generation dataset. Missions target public live-web data, so exact values can drift between runs; checks allow for that where it matters.