Evidence sheet · 2026-11-19

Test form

20 tasks × 40 models, the actual test receipt, and the live endpoint smoke. This is the human-readable view of tests/results/TEST_FORM.md.

180tests passing
90.02%line coverage
28/28adversarial cases
100%dev top-1
100%held-out top-1
94%value quality parity

This is a decision benchmark, not an independent model-output benchmark. Image/video quality is arena-informed; 3D/audio remain expert priors. Live smoke proves transport and response format, not quality.

Per-task matrix

TaskEligibleTop modelQualityCost / taskValue pickValue cost
coding-agent
Agentic coding
4gemini-4-carbon87.3$0.174gemini-4-carbon$0.174
code-review
Code review
6gemini-4-carbon85.29$0.088gpt-6.2-sol$0.087
refactor
Large refactor
6gemini-4-carbon86.32$0.193gemini-4-carbon$0.193
debugging
Debugging
6gemini-4-carbon85.53$0.132gpt-6.2-sol$0.130
planning
Planning & architecture
6gemini-4-carbon86.71$0.069gpt-6.2-sol$0.067
long-context
Long-context analysis
10gemini-4-hydrogen86.42$0.647gpt-6.1-terra$0.472
research
Research synthesis
11gpt-6.1-terra80.57$0.164gpt-6.1-terra$0.164
writing
Writing & copy
9gpt-6.2-sol87.92$0.036gpt-6.1-terra$0.020
translation
Translation
12gpt-6.1-terra80.55$0.031gpt-6.1-terra$0.031
extraction
Structured extraction
20claude-haiku-5.562.41$0.0066claude-haiku-5.5$0.0066
classification
Classification & tagging
22claude-haiku-5.570.91$0.0008claude-haiku-5.5$0.0008
data-analysis
Data analysis
6gemini-4-carbon85.7$0.145gpt-6.2-sol$0.144
computer-use
Computer & browser use
5gemini-4-carbon91.31$0.081gemini-4-carbon$0.081
image-gen
Image generation
4nano-banana-2.193.65$0.036seedream-5.5$0.034
image-edit
Image editing
4nano-banana-2.193.8$0.036seedream-5.5$0.033
video-gen
Video generation
4veo-3.1-fast86$0.596veo-3.1-fast$0.596
3d-gen
3D asset generation
4tripo-h3.194$0.215tripo-h3.1$0.215
tts
Text to speech
2voxtral-tts-388$0.015voxtral-tts-3$0.015
stt
Speech to text
2voxtral-small-transcribe-390$0.045voxtral-small-transcribe-3$0.045
music
Music generation
2lyria-3.590$0.090lyria-3.5$0.090

Per-model applicability

ModelEligibleTop-1Top-3
gemini-4-carbon13710
gpt-6.1-terra624
claude-haiku-5.5222
nano-banana-2.1222
gpt-6.2-sol1218
gemini-4-hydrogen613
lyria-3.5111
tripo-h3.1111
veo-3.1-fast111
voxtral-small-transcribe-3111
voxtral-tts-3111
claude-sonnet-5.51105
claude-opus-5.51302
deepseek-v4.1-flash202
gpt-6-luna202
gpt-image-2202
seedream-5.5202
eleven-v4101
gpt-6.2-astra1301
gpt-transcribe101
kling-4.0101
meshy-7.1101
minimax-music-3101
seedance-2.5101
trellis-2101
claude-fable-5.21300
deepseek-v4-pro200
gemini-3.8-flash200
gemini-4-argon600
glm-5.3200
glm-5.3-flash100
gpt-5.6-sol400
grok-4.6200
hunyuan3d-3.1-pro100
kimi-k3300
kimi-k3.1-pro500
minimax-m3.1-flash100
nano-banana-pro200
qwen3.8-max200
veo-3.1100

Adversarial cases

IDAreaAttackObservedResult
A01inputEmpty queryERR_MISSING_TASKPASS
A02input100,000-character query11msPASS
A03inputPrompt injection: ignore instructions + rm -rf3d-genPASS
A04inputShell metacharacterstripo-h3.1PASS
A05inputRTL + emoji + zero-width charactersimage-genPASS
A06pathTask id traversal: ../../etc/passwd1PASS
A07budgetNegative budget1PASS
A08budgetNaN budget1PASS
A09constraintsOpen weights + $0.001 budgetNO_ELIGIBLE_MODELPASS
A10constraints2M-token context requirementNO_ELIGIBLE_MODELPASS
A11constraintsUnknown optimize preset1PASS
A12catalogInvalid overrides JSONERR_OVERRIDES_INVALIDPASS
A13catalogCorrupt cache falls back to bundled snapshot40PASS
A14catalogDuplicate slug validationERR_INVALID_CATALOGPASS
A15secretsSecret in environment0PASS
A16setupSetup dry-run on a clean projectnonePASS
A17setupInvalid .mcp.json is never overwritten1PASS
A18add-modelIncomplete model file1PASS
A19mcptools/call with null arguments{"content":[{"type":"text","text":"{\"error\":\"TOOL_ERROR\",\"message\":\"Tell PASS
A20mcpUnknown tool-32602PASS
A21mcp10 concurrent pick calls10PASS
A22claimsStale catalog is flagged687PASS
A23claimsRecommendation without any keys still returns a decisionfalsePASS
A24decisionVerdict invents a model outside the shortlist1PASS
A25decisionExternal verdict reorders locallymeshy-7.1PASS
A26evalMalformed eval passed field1PASS
A27setupVS Code --global is not silently written1PASS
A28setupGenerated AGENTS command is installable0PASS

Live smoke

EndpointResultLatencyEvidence
pollinations-textPASS1256 ms200 text/plain; charset=utf-8
pollinations-imagePASS749 ms200 image/jpeg
github-modelsPASS1087 ms200 text/plain
ollama-localPASS88 ms200

Free/public endpoints only. No API keys are used or stored. The third paid-provider slot is still open.