How to read vendor AI benchmark claims, and test them yourself

By Bot Tutorials · Published Oct 29, 2026 · Updated Oct 29, 2026 · Facts last verified Oct 29, 2026

A method for reading the benchmark and cost claims in AI launch posts, using October 2026 announcements as examples, plus a 10-task evaluation you can run in an afternoon.

1. Why this matters

Every launch post comes with scores. They are real measurements, but on the vendor's choice of tasks, settings and comparison models. A user's question is narrower: does it do my tasks better or more cheaply?

2. Four checks for any claim

1. Compared with what? OpenAI compares GPT-6.1 Terra mainly with its predecessor GPT-5.6 Terra and its own siblings. Anthropic compares Haiku 5.5's cost with Haiku 4.5. A comparison with an older model doesn't tell you where it stands against rivals.
2. At what setting? OpenAI quotes Terra at “medium” and “max” reasoning effort. Your product may run a different default (OpenAI says some Plus and Pro users see Terra at Medium in Work).
3. Whose benchmark? Some tests are public (Terminal-Bench, OSWorld, MMMU-Pro) and some are internal. OpenAI's hallucination figures (53.6% to 16.7% at low effort) come from “our internal benchmark of error-inducing prompts”, which nobody outside can reproduce.
4. Per token or per task? A model that is cheap per token can still cost more per task if it uses more tokens. OpenAI makes this point when comparing Terra's cost per task with Claude Opus 5.5.

3. A 10-task evaluation

1. Collect 10 real tasks with known good answers.
2. Write pass/fail criteria before you run anything.
3. Run each task 3 times per model, because outputs vary between runs.
4. Record pass rate, time, and cost from the usage fields.
5. Choose on your own results, not on the vendor's chart.

# log one row per run
model,task_id,run,pass,seconds,input_tokens,output_tokens
gpt-6.1-terra,t01,1,1,4.2,812,233

4. Limitations

Ten tasks catches big differences, not small ones. Use more tasks for decisions that cost real money.