Index

A Benchmark Table Is Not an Eval

· Tom Cranstoun · 7 min read

When Google opened Gemini 4 Argon to a small group of testers on 30 September, it came with a table. Argon led Claude Opus 5.5 on 14 of 18 benchmarks, and Google was honest enough to show the four it lost. Through 30 November, Argon costs half what Opus does. On 1 December the price rises to match.

So the question for anyone paying for a model this autumn is simple. Is Argon better, or just cheaper for six weeks?

Jessica Wachtel at The New Stack didn't take the table's word for it. She built three hard tests from categories where Google claimed a lead. She wrote and checked the answer keys before either model saw a task, and a script did the grading. Her results don't look much like the table.

What She Found

The bug hunt was a tie. A 3,500-line Python library, 14 planted logic bugs, five decoys. Both models found all 14, including four that only show up when you follow a value across three or more files. Neither flagged a decoy. Both fixed every bug without breaking any of the 93 existing tests. Argon did it for less than half the cost, and would still be cheaper at full price because it used fewer tokens.

The billing spec was a rout, over one line. A subscription billing engine built from a written spec, graded by 80 hidden tests on daylight saving, month-end dates and rounding. Opus passed 80 of 80. Argon passed 11. Its code passed an extra argument to Python's datetime.combine(), which raises an error, and most of the module leaned on that helper. When she fixed that one line by hand, Argon scored 80 of 80. She kept the official score and told readers why it was so low.

The long-context test was the worrying one. A made-up wiki of 400 services, with a question about blast radius: which services go down when one fails. The answer is 33. Opus named all 33 at the first size and 32 at the next, with nothing wrong. At the first size Argon got 20 correct, plus 8 that don't belong. At the next it managed 19, with 16 invented. It treated a dependency as removed after the wiki had put it back. It also answered in a few hundred tokens where Opus reasoned through more than 5,000 first. She suspects the speed and the errors are linked, and says plainly that one run can't prove it.

Her verdict was that she wouldn't trust Argon unchecked on any task.

The Table and the Tests Aren't Arguing

It's tempting to read this as "the benchmarks lied". I don't think they did. A benchmark tells you how a model does on that particular set of tasks, run the way the vendor ran it. It's a real measurement of something. It just isn't a measurement of your work.

Your work has its own shape. Maybe it's billing code where one wrong helper breaks everything downstream. Maybe it's a long document where a reinstated dependency matters more than any single fact. A table of other people's tasks can't tell you how a model handles those. Only a test built from your own work can.

That's what an eval is: a fixed task, a known right answer, and a model put to it cold. The point isn't to crown a winner. It's to find out what this model does with this kind of work before you hand it the real thing.

What Made Her Test Worth Reading

What I liked most about the article was the method showing through, more than the verdict.

  • The key came first. She built and checked her answer keys before testing either model. A key written after you've seen the answers drifts towards them, however careful you are.
  • A script did the marking. Nobody's mood or fondness for a vendor touched the scores.
  • She reported the cause, not just the score. "11 of 80" on its own reads as a model that doesn't understand billing. The truth was a model that understood it and broke one helper. Both facts matter, and she gave both.
  • She said what one run can't prove. That sentence is rarer in model comparisons than it should be.

What a Reader Still Can't See

Even a careful write-up leaves gaps, and they're the same gaps in almost every comparison I read.

  • How many runs. One per model. A tie on one run tells you both models can find every bug, not how often they will.
  • Which version. "Gemini 4 Argon" and "Claude Opus 5.5" are names. A vendor can point a name at a different model later, and you won't be told.
  • What a token is. Her 500K document came to 549,483 tokens on Argon and 784,087 on Opus. A bigger one came to 1.4 million on Claude's tokenizer and was simply rejected. Price per token compares nothing across two vendors. Price per finished task does.
  • What the score leaves out. "20 of 33" sounds like a partial answer. With 8 invented services beside it, it's a wrong one. A score with no count of wrong answers hides exactly the failure that would hurt you.
  • When the price changes. Every cost comparison in the piece has a shelf life of six weeks.

None of that's a criticism of her. She did more than most. It's that there's no agreed place to write these things down, so they live in paragraphs, and a machine reading the paragraphs has to guess.

So I Wrote Them Down

MX already has profiles: named lists of the fields each kind of document needs. I've added one for evals. An eval record holds the result beside the facts a reader needs to weigh it:

type: eval-record
mx:
  evalTask: "Answer graph questions over a large wiki with planted traps."
  evalSubjects:
    - model: example-model-4
      provider: example-lab
      route: openrouter
      settings: default reasoning
  evalGrading: script
  evalRuns: 1
  evalKey:
    fixedAt: 2026-10-12
  evalResults:
    - model: example-model-4
      score: 20
      outOf: 33
      wrong: 8

Four of those fields are required: the task, the models and their settings, how the answers were graded, and how many runs. Without any one of them, a score can't be read. With all four, a thin test still shows its limits. One run is a valid value. It just can't pass itself off as a rate any more.

To check the profile against something real, I recorded her three tests as eval records. They hold up well. The gaps they expose are the ones above: no versions, no key dates, one run each. That's the profile doing its job.

The profile is a proposal to The Gathering, the community that looks after the MX standard, and it isn't ratified. The field definitions are in the published dictionary at https://mx.allabout.network/canon/fields-data.yaml.

What I'd Pass On

If you're weighing Argon's half-price window, or any model choice, six weeks is long enough to run your own evals.

  • Take three to five tasks from your real work, the ones where a wrong answer costs you something.
  • Write and check the answer key before any model sees the task, and date it.
  • Let a script mark it where you can. Where you can't, say who marked it.
  • Run each task more than once. If you only run it once, say so.
  • Count wrong answers as well as right ones.
  • Compare cost per finished task, not per token.
  • Rerun when the model changes, including when the name doesn't.

Then write down how you ran it, next to what you found. A score with no method is a rumour. A score with its method is evidence the next reader, person or machine, can check.

A declared interest. I wrote the books on MX and consult on it, so I have a stake in the practice this post recommends. I drafted it with help from Claude, made by Anthropic, whose model is one of the two in the test. Every figure here is Jessica Wachtel's, from the article below; I haven't rerun her tests.

Source

Wachtel, J. (2026, October 18). Gemini 4 Argon vs. Claude Opus 5.5: Half the price and about half as accurate. The New Stack. https://thenewstack.io/gemini-argon-claude-opus/