Evidence sheet · 2026-11-19
Test form
20 tasks × 40 models, the actual test receipt, and the live endpoint smoke. This is the human-readable view of tests/results/TEST_FORM.md.
180tests passing
90.02%line coverage
28/28adversarial cases
100%dev top-1
100%held-out top-1
94%value quality parity
This is a decision benchmark, not an independent model-output benchmark. Image/video quality is arena-informed; 3D/audio remain expert priors. Live smoke proves transport and response format, not quality.
Per-task matrix
| Task | Eligible | Top model | Quality | Cost / task | Value pick | Value cost |
|---|---|---|---|---|---|---|
| coding-agent Agentic coding | 4 | gemini-4-carbon | 87.3 | $0.174 | gemini-4-carbon | $0.174 |
| code-review Code review | 6 | gemini-4-carbon | 85.29 | $0.088 | gpt-6.2-sol | $0.087 |
| refactor Large refactor | 6 | gemini-4-carbon | 86.32 | $0.193 | gemini-4-carbon | $0.193 |
| debugging Debugging | 6 | gemini-4-carbon | 85.53 | $0.132 | gpt-6.2-sol | $0.130 |
| planning Planning & architecture | 6 | gemini-4-carbon | 86.71 | $0.069 | gpt-6.2-sol | $0.067 |
| long-context Long-context analysis | 10 | gemini-4-hydrogen | 86.42 | $0.647 | gpt-6.1-terra | $0.472 |
| research Research synthesis | 11 | gpt-6.1-terra | 80.57 | $0.164 | gpt-6.1-terra | $0.164 |
| writing Writing & copy | 9 | gpt-6.2-sol | 87.92 | $0.036 | gpt-6.1-terra | $0.020 |
| translation Translation | 12 | gpt-6.1-terra | 80.55 | $0.031 | gpt-6.1-terra | $0.031 |
| extraction Structured extraction | 20 | claude-haiku-5.5 | 62.41 | $0.0066 | claude-haiku-5.5 | $0.0066 |
| classification Classification & tagging | 22 | claude-haiku-5.5 | 70.91 | $0.0008 | claude-haiku-5.5 | $0.0008 |
| data-analysis Data analysis | 6 | gemini-4-carbon | 85.7 | $0.145 | gpt-6.2-sol | $0.144 |
| computer-use Computer & browser use | 5 | gemini-4-carbon | 91.31 | $0.081 | gemini-4-carbon | $0.081 |
| image-gen Image generation | 4 | nano-banana-2.1 | 93.65 | $0.036 | seedream-5.5 | $0.034 |
| image-edit Image editing | 4 | nano-banana-2.1 | 93.8 | $0.036 | seedream-5.5 | $0.033 |
| video-gen Video generation | 4 | veo-3.1-fast | 86 | $0.596 | veo-3.1-fast | $0.596 |
| 3d-gen 3D asset generation | 4 | tripo-h3.1 | 94 | $0.215 | tripo-h3.1 | $0.215 |
| tts Text to speech | 2 | voxtral-tts-3 | 88 | $0.015 | voxtral-tts-3 | $0.015 |
| stt Speech to text | 2 | voxtral-small-transcribe-3 | 90 | $0.045 | voxtral-small-transcribe-3 | $0.045 |
| music Music generation | 2 | lyria-3.5 | 90 | $0.090 | lyria-3.5 | $0.090 |
Per-model applicability
| Model | Eligible | Top-1 | Top-3 |
|---|---|---|---|
| gemini-4-carbon | 13 | 7 | 10 |
| gpt-6.1-terra | 6 | 2 | 4 |
| claude-haiku-5.5 | 2 | 2 | 2 |
| nano-banana-2.1 | 2 | 2 | 2 |
| gpt-6.2-sol | 12 | 1 | 8 |
| gemini-4-hydrogen | 6 | 1 | 3 |
| lyria-3.5 | 1 | 1 | 1 |
| tripo-h3.1 | 1 | 1 | 1 |
| veo-3.1-fast | 1 | 1 | 1 |
| voxtral-small-transcribe-3 | 1 | 1 | 1 |
| voxtral-tts-3 | 1 | 1 | 1 |
| claude-sonnet-5.5 | 11 | 0 | 5 |
| claude-opus-5.5 | 13 | 0 | 2 |
| deepseek-v4.1-flash | 2 | 0 | 2 |
| gpt-6-luna | 2 | 0 | 2 |
| gpt-image-2 | 2 | 0 | 2 |
| seedream-5.5 | 2 | 0 | 2 |
| eleven-v4 | 1 | 0 | 1 |
| gpt-6.2-astra | 13 | 0 | 1 |
| gpt-transcribe | 1 | 0 | 1 |
| kling-4.0 | 1 | 0 | 1 |
| meshy-7.1 | 1 | 0 | 1 |
| minimax-music-3 | 1 | 0 | 1 |
| seedance-2.5 | 1 | 0 | 1 |
| trellis-2 | 1 | 0 | 1 |
| claude-fable-5.2 | 13 | 0 | 0 |
| deepseek-v4-pro | 2 | 0 | 0 |
| gemini-3.8-flash | 2 | 0 | 0 |
| gemini-4-argon | 6 | 0 | 0 |
| glm-5.3 | 2 | 0 | 0 |
| glm-5.3-flash | 1 | 0 | 0 |
| gpt-5.6-sol | 4 | 0 | 0 |
| grok-4.6 | 2 | 0 | 0 |
| hunyuan3d-3.1-pro | 1 | 0 | 0 |
| kimi-k3 | 3 | 0 | 0 |
| kimi-k3.1-pro | 5 | 0 | 0 |
| minimax-m3.1-flash | 1 | 0 | 0 |
| nano-banana-pro | 2 | 0 | 0 |
| qwen3.8-max | 2 | 0 | 0 |
| veo-3.1 | 1 | 0 | 0 |
Adversarial cases
| ID | Area | Attack | Observed | Result |
|---|---|---|---|---|
| A01 | input | Empty query | ERR_MISSING_TASK | PASS |
| A02 | input | 100,000-character query | 11ms | PASS |
| A03 | input | Prompt injection: ignore instructions + rm -rf | 3d-gen | PASS |
| A04 | input | Shell metacharacters | tripo-h3.1 | PASS |
| A05 | input | RTL + emoji + zero-width characters | image-gen | PASS |
| A06 | path | Task id traversal: ../../etc/passwd | 1 | PASS |
| A07 | budget | Negative budget | 1 | PASS |
| A08 | budget | NaN budget | 1 | PASS |
| A09 | constraints | Open weights + $0.001 budget | NO_ELIGIBLE_MODEL | PASS |
| A10 | constraints | 2M-token context requirement | NO_ELIGIBLE_MODEL | PASS |
| A11 | constraints | Unknown optimize preset | 1 | PASS |
| A12 | catalog | Invalid overrides JSON | ERR_OVERRIDES_INVALID | PASS |
| A13 | catalog | Corrupt cache falls back to bundled snapshot | 40 | PASS |
| A14 | catalog | Duplicate slug validation | ERR_INVALID_CATALOG | PASS |
| A15 | secrets | Secret in environment | 0 | PASS |
| A16 | setup | Setup dry-run on a clean project | none | PASS |
| A17 | setup | Invalid .mcp.json is never overwritten | 1 | PASS |
| A18 | add-model | Incomplete model file | 1 | PASS |
| A19 | mcp | tools/call with null arguments | {"content":[{"type":"text","text":"{\"error\":\"TOOL_ERROR\",\"message\":\"Tell | PASS |
| A20 | mcp | Unknown tool | -32602 | PASS |
| A21 | mcp | 10 concurrent pick calls | 10 | PASS |
| A22 | claims | Stale catalog is flagged | 687 | PASS |
| A23 | claims | Recommendation without any keys still returns a decision | false | PASS |
| A24 | decision | Verdict invents a model outside the shortlist | 1 | PASS |
| A25 | decision | External verdict reorders locally | meshy-7.1 | PASS |
| A26 | eval | Malformed eval passed field | 1 | PASS |
| A27 | setup | VS Code --global is not silently written | 1 | PASS |
| A28 | setup | Generated AGENTS command is installable | 0 | PASS |
Live smoke
| Endpoint | Result | Latency | Evidence |
|---|---|---|---|
| pollinations-text | PASS | 1256 ms | 200 text/plain; charset=utf-8 |
| pollinations-image | PASS | 749 ms | 200 image/jpeg |
| github-models | PASS | 1087 ms | 200 text/plain |
| ollama-local | PASS | 88 ms | 200 |
Free/public endpoints only. No API keys are used or stored. The third paid-provider slot is still open.