I Built a Decider and Measured It

I wanted to know whether to build a decider, buy one, or fix the files it reads. So I built one on my laptop, called TypeSafe's hosted Jev, tried the model Apple ships on recent Macs, and put the same questions to all of them, with and without MX metadata in the files.

The files mattered more than the model. On the questions where each file appeared both bare and carrying MX, the metadata cut made-up answers by roughly half to three-quarters on every model that made them up. My 3B laptop model went from inventing an answer on 44 of 64 items to 20. My 70B one went from 8 to 2. Hosted Jev invented nothing either way. On bare files that didn't state the answer, it said "I can't tell"; with MX, it gave the right one.

So my answer is MX first, then buy or build. Buy when the decision acts on its own. Build when nothing may leave the machine, and check what it says.

The Short Version

• MX cut made-up answers. A made-up answer is a wrong one that asserts something, an owner, a status or a verdict, instead of saying it can't tell. With MX, my 3B rebuild fell from 69% to 31%, my 8B from 41% to 14%, my 70B from 12.5% to 3.1%, and Apple's on-device model from 30% to 17%.
• Hosted Jev made none up. Without MX it said "I can't tell" and meant it. With MX it answered, and got it right. Across all 354 questions it missed 15, and only once was it sure of a wrong answer.
• A decider is fast and cheap. Scoring the first token of a small local model gives a decision in about a tenth of a second, for nothing.
• Being sure and right is what you pay for. When hosted Jev was sure, it was wrong about once in every 130 answers. My best rebuild was wrong about once in 10.
• Apple's on-device model can't be a decider yet. It doesn't show its confidence, and when I made it vote, it was sure when wrong.
• The fields do the work. Structured fields produced the result. Plain-language prose helped, but on its own it was the weakest input.
• Two gaps remain. Small deciders rarely use a "none of the above" option, and their probabilities drift slightly between runs even when their decisions don't.

If you want the wider argument about why this matters, I've set it out in Nobody Got Fired for Using GEO.

What I Built

In September 2026 TypeSafe released Jev, a model that never writes a word. The launch post says it "gives up string generation", and the primitives documentation says it "returns a probability distribution over your options or levels, never a value outside them". Avi Chawla then published a walkthrough, "Build your own Jev (100% local)", on 20 September, for building the same interface on an open model, so I built one with Ollama, which I already use.

The mechanism fits in a paragraph. A language model reading a prompt produces one vector for the next position, with a score for every token it knows. Normal generation samples from that vector and goes round again. A decision only needs the first vector. Label each permitted answer with a single letter and put its meaning in the prompt. Ask Ollama for the top twenty logprobs of the first token, keep the ones that are your labels, and normalise across them. That's one forward pass, one token, temperature zero and a fixed seed.

It came to about sixty lines added to my existing local model client, plus a unit test. It copies Jev's inference path, not its calibration training.

How I Tested It

I wrote twenty hand-labelled cases for each of five questions: support ticket routing, change-impact triage for documentation, internal link relevance, content type, and whether a document is current. The sets are small, so treat every accuracy here as indicative rather than a benchmark.

Three lanes saw the same prompt: the decider on three local models, the same 3B model asked to write its answer, and Claude haiku, called from an empty directory so it received the question and nothing else. Every number comes from one run file, rendered by a script.

Speed Holds

The 3B decider answers in 102 ms at the median. The same model writing a letter and a sentence takes 441 ms. Claude takes about five seconds and $0.0027 a decision, and it's the most accurate lane by a distance. That hosted price won't stay put: Epoch AI measures the cost of a given level of AI performance falling about 13 times a year. I've compared that curve with Moore's law and with this decider.

The reasoning model, gpt-oss:20b, couldn't be scored: its first token opens a thinking channel, so no label appears. This technique needs a model whose first token is the answer.

The decider is fast and cheap, then, but on these general questions the small models were well short of Claude.

The Main Result: Embedded MX

MX, short for machine experience, is the practice of making a file explain itself to an agent that meets it in isolation. It says the metadata travels with the file, so I tested it that way. I took ten documents and made each into two PDFs.

• The bare PDF holds the page text and nothing else. The text carries no date, no status and no owner, on purpose.
• The MX PDF is the same file with an MX packet written into its XMP metadata, the standard place a PDF stores information about itself. The packet holds status, dates, an expiry or review date, what supersedes the document, and who owns it. It also holds a short summary and instructions for an agent.

One script reads every PDF the same way: take whatever the packet declares, extract the page text, and pass both to the model, declared fields first. On a bare file the packet is empty, so the model gets the text alone. Nobody types anything; the file decides what the model sees.

On the bare PDFs, the 3B decider guessed an answer either way for eight of ten documents that say neither. Asked whether a bare file names an owner, it said yes nine times out of ten, at confidence up to 0.86, for files that name nobody. That's a small model inventing accountability for a refund policy.

With the packet embedded, the same model went to 100% on both questions. On currency, every local lane reached 100%. The gap between the laptop and Claude closed.

Claude got every case right on both versions, because on a bare file it says "cannot be determined" and means it. The small models needed telling, and the embedded metadata told them.

What in the Packet Did the Work

I expected the plain-language summary to matter most, so I tested it. Two more versions of the packet went to the small models: fields only, and prose only.

My guess was wrong. With the prose removed, the 3B decider still scored 100% on both questions, with lower confidence. The labelled fields are what it reads.

Prose on its own was the weakest input. "Reviewed by finance operations" isn't read as an owner the way author: finance-ops is, and the 3B generator's accountability score fell to 30%. A sentence that points at a field also fails when its target is missing: with expires removed, the 8B decider called a policy "not current" at 0.965, because the summary said "valid until the expiry date declared here".

The prose still helped. Without the sentence saying the document was in force, the 8B decider read status: active and a future expires date and called two documents not current. The full packet did best. The fields are the facts, the prose is the reading of them, and a small model uses whichever it can parse.

Pasting Metadata Is Not the Same

Before the PDF test, I tried pasting frontmatter into the prompt, on a separate set of plain-text documents, so the starting scores differ from the PDF tables. The 8B decider went from 50% to 100% on currency, with confidence up from 0.794 to 0.983. The 3B models got worse: the decider misread status: active and dropped from 80% to 60%, and the generator ignored the frontmatter altogether and fell from 90% to 40%.

The PDF test changed two things: the metadata arrived as labelled lines under a heading naming the packet as its source, and it came with a summary written for a reader. Both 3B lanes went to 100%. How the metadata is presented matters to a small model, which is an argument for a standard packet read by one shared script rather than ad hoc text in a prompt.

Two Things That Still Go Wrong

The escape option goes unused. A restricted softmax always sums to one, so it can't say "none of these" unless "none" is a declared label. I declared one wherever cases could fall outside the list. The small deciders rarely chose it. Three support tickets were meant to escape: a leaked customer list, a legal request for everything held on a person, and a user receiving threats. Both deciders routed all three to "account access", the 8B at 0.82, 0.97 and 0.95. The 8B also decided a pull request titled "Misc fixes" needed no documentation update, at 0.997. Declaring the escape label is necessary, and it isn't enough: measure whether it gets used.

Probabilities drift; decisions don't. I'd written "deterministic by construction" in a code comment, so I measured it. Across three runs of every case, every decider made the same decision 100 times out of 100. The probabilities moved slightly, by at most 0.008493 on the 3B model, because the first pass ran with a partial prompt cache and the repeats with a full one. It's reproducible given the cache state, not bit-identical across it. That's a fine property, and it isn't the one I'd claimed.

A Second Witness: Token Confidence

In 2025 I wrote up a framework for evaluating AI confidence, which reads the probability behind each word a model writes. I added it to the generation lane.

The probability of the answer letter behaves like the decider's confidence. On every question it was higher when the model was right than when it was wrong, by ten to twenty-seven points. On PDF currency it was 72.5% when right and 45.9% when wrong, against the decider's 77.6% and 49.5%.

It also confirmed the carrier result. On PDF currency, the generator's confidence in its answer rose from 0.456 on the bare files to 0.781 on the declared ones, as its accuracy went from 20% to 100%. With pasted frontmatter it stayed at 0.606 and 0.607 while accuracy fell. A declaration the model reads shows up in the number, and one it ignores doesn't.

The framework's whole-reply score didn't track the decisions, because the explanation sentence hedges whether the label is right or not. A decision's confidence is one token's probability, and the rest is the model talking.

Measuring Jev Itself

Everything above measures my rebuild. On 2026-10-09 I ran TypeSafe's hosted Jev through 354 hand-labelled cases in 17 decision evals, with the questions in this post and a 20-case confirmation eval among them. The model was jev-1.13.0, called as jev-latest through TypeSafe's System One API. These sets are small too, so treat the figures as indicative.

Hosted Jev scored 95.8% across the 354 decisions, at a median of 264 ms a decision and $0.0070 for all 354. It led, or tied for the lead, on 16 of 17 evals. My 8B rebuild on llama3 scored 74.6%. The bigger difference is in the confidence. Hosted Jev gave 267 answers at 0.9 or above and 2 of them were wrong. Its mean confidence was 0.910 when right and 0.500 when wrong. The 8B rebuild gave 248 answers at 0.9 or above, and 42 of them were wrong. Hosted Jev drifts too: 1 decision in 354 changed between two identical calls, and its probabilities move by up to 0.15.

The confirmation eval shows the gap most plainly. A decider gets a task and another model's answer, and decides whether to confirm it. Of the 20 cases, 10 are bare, so the right answer is "cannot be determined", and 10 declare a scope and a corpus. Hosted Jev got 20 of 20. It chose "cannot be determined" on all 10 bare cases, held all 7 invalid declared answers and confirmed the 3 valid ones. Both my rebuilds scored 25% and never chose "cannot be determined". Each confirmed 3 of the 7 invalid answers, and the 8B one was more confident when wrong (93.3%) than when right (86.6%). Claude Haiku scored 95% at a median of 6.4 seconds, against hosted Jev's 262 ms.

I read that gap as what calibration training buys, and it's the part my sixty lines don't copy. Air-gapped, my strongest rebuild runs on llama3 70B and scores 87.9%. The best local option for each eval, picked after the fact as an upper bound, reaches 89.0%, against hosted Jev's 95.8%. A bar of 0.9 works for hosted Jev. It works for none of my local rebuilds. The 70B and 8B are wrong too often at it, and the 3B rarely reaches it. So my tools pick the decider to suit where they run: hosted Jev on a machine that may reach the internet and holds a key, the largest local one it can run when it's air-gapped. A hosted call that fails is answered locally. Declared metadata still matters to hosted Jev: on the bare documents the right call was "cannot be determined", and only the declared ones let it reach a verdict.

Those 2026-10-09 figures came from an exploratory run. On 2026-10-15 I ran hosted Jev again under a registration I wrote before the first call, and the main findings held. Its calibration error was 0.027 against the 0.10 ceiling I'd set, and it chose "cannot be determined" on 76 of the 78 cases that called for it. At 0.9 it answered 188 of its non-escape decisions and got one wrong; my 8B rebuild answered 208 and missed 37. Then the bar met cases it had never seen. Fitted at 0.9 on thirty refund messages, it answered 21 of a fresh set at that bar, with none wrong. That's a small sample, but it's the first time one of my bars has held on unseen data. Two days later I repeated the registered run on a second machine, a Mac Studio, and got 96.3%, with a calibration error of 0.035 and 77 of the 78 "cannot be determined" cases caught.

The Model Already on a Mac

Recent Macs come with a small Apple model built in, the on-device Foundation Model, and Apple's fm serve tool lets other programs on the machine ask it questions. It costs nothing, keeps the data on the machine and needs no setup, so it looked like the obvious air-gapped decider. I put the same 354 cases to it.

It can't be a decider in the sense this post means. A decider reads how strongly the model leans towards each answer, and Apple's doesn't share that. It only gives back text. So I tried two other ways of asking it.

The first was to ask for the answer letter alone, once. That scored 69.2%, at a median of 644 ms a decision. Asked for the letter and a sentence of reasoning, it did worse, at 55.9%. Asking for less got more right.

The second was to ask the same question ten times and count the votes, to get something that looks like confidence. It scored about the same, 69.5%, took ten times as long, and the votes weren't honest. When nine or ten out of ten agreed, the answer was still wrong 57 times in 270. Hosted Jev, at the same level of certainty, was wrong twice.

So the Mac's own model is a reasonable guesser and a poor judge of its own guesses. If you use it, ask once for the letter alone, and don't treat any reply as sure.

On 2026-10-17 I ran it again on a Mac Studio, on 620 cases this time, with the pipeline's questions and real pages from this site added. On the 294 cases both runs share, it scored within two points of its first result, so the picture held. The new cases found something else. Asked who a page is written for, it returned "refused: safety guardrail" on 16 of 20 pages when asked once, and refused all ten votes on 11 of 20. That wasn't confined to children's pages. Asked once, it refused 4 children's pages out of 5, and 12 others out of 15. By vote it answered every children's page correctly and refused business, adult and unclear ones. The guardrail seems to react to the question rather than the page, and not the same way twice. Whatever the cause, a tool that won't answer the question can't be a step in a pipeline. Its sure answers were no better than before: at nine or ten votes out of ten it was wrong 93 times in 488, still about one in five.

Asking It a Pipeline's Own Questions

A cheap decider tempts you to hand it every bounded question you have. One of my production pipelines asks a generating model three of them: who a page is written for, whether a chart's alt text actually states the numbers, and if a report section stays in one voice. Each is multiple choice with an escape, so it's the decider's kind of question. On 2026-10-14 I wrote twenty cases for each from the pipeline's own rubric, registered the bar before running anything, and ran them on the local rebuilds.

None of the three cleared it. The 8B decider got the audience question right 80% of the time, but it never once said "indeterminate": a loading placeholder and a login form were "normal" at 0.86 and above, and it was as confident when wrong (0.956) as when correct. On the alt-text question it held the bar, one wrong among thirteen answers at 0.9 or above, and still failed on accuracy, because every miss was the same one: "bar chart showing revenue growing" judged adequate. The rubric's one rule is that describing the picture isn't stating its data, and the decider didn't apply it. The voice question it got right 60% of the time and called mixed sections clean.

On 2026-10-18 I put the same sixty cases to hosted Jev, under a new registration with the same bars. It cleared accuracy and the 0.9 bar on all three questions. It got the alt-text question right every time, and of its 50 sure answers across the three sets, one was wrong. It missed one bar, and that miss turned out to be the more useful result. On the audience question it said "indeterminate" on only one of the three pages that needed it. Claude Haiku made exactly the same two mistakes on the same two pages.

When two unrelated models get the same cases wrong in the same way, look at the question. My rubric's label for the ordinary case asked only that nothing pointed to a restricted audience, so a page with no evidence at all qualified. I changed the rubric to need positive evidence of a normal audience, and wrote a harder set with ten indeterminate pages, registered before the run. Hosted Jev got all thirty right; Haiku got twenty-nine. Two caveats apply. One of the two shared mistakes went away because I changed its answer key, not because a model changed its mind, and I wrote most of the new pages after seeing what failed. The rubric change itself is live, though, so every model that answers the question now gets a better one.

So the cheap path is open for these questions, but I haven't taken it yet. A hosted decider adds a supplier to every job, and that's a business call rather than a technical one. The case sets stay, because they're what says when that changes. The cheap path is earned per question, by a measurement written down before the run, and sometimes that test finds a fault in the wording.

Build, Buy or MX?

I've now run every option I could get to on the same 354 questions. Three things decide between them: how often each one is right, whether its sure answers can be trusted, and how much it makes up when the file doesn't say.

Being Sure and Right

A decider answers two questions: what's the answer, and how sure am I? When it's sure, you let it act without a person checking. When it isn't, it passes the case to someone else. So the measure that matters most is how often it's wrong when it says it's sure.

• Hosted Jev: right 95.8% of the time, and wrong about once in every 130 sure answers. A quarter of a second a decision, and under a cent for all 354.
• My rebuild on a large local model (70B): right 87.9% of the time, wrong about once in every 10 sure answers. Three seconds a decision, and it needs a machine with 64 GB of memory.
• My rebuild on a mid-sized local model (8B): right 74.6% of the time, wrong about once in every 6 sure answers.
• Apple's on-device model: right about 69% of the time, wrong about once in every 5 sure answers.

Hosted Jev's mistakes were honest ones, too. It got 15 of the 354 wrong. Five times it said "I can't tell" when the answer was there. Nine more times it gave a wrong answer but wasn't sure enough to act, so a decider would pass the case to someone else. It was sure of a wrong answer once. When it was wrong, it nearly always said it didn't know.

TypeSafe trained Jev to close that gap. My sixty lines copy the speed, not the training, and the training is the difference between 1 in 130 and 1 in 10.

Made-Up Answers, With and Without MX

The paired evals put the same page or file to each decider twice, bare and then carrying MX metadata. Each decider saw 64 bare items and the same 64 with MX. I counted the made-up answers: wrong answers that asserted something, an owner, a status or a verdict, where the right reply was "I can't tell" or a different fact.

• The 3B rebuild: from 44 made-up answers to 20, which is 55% fewer.
• The 8B rebuild: from 26 to 9, 65% fewer.
• The 70B rebuild: from 8 to 2, 75% fewer.
• Apple's on-device model, asked once: from 19 to 11, 42% fewer.
• Hosted Jev: none either way.

MX did something different for each. For the small models, it stopped the guessing. On whether a PDF is current, the 3B rebuild went from 20% to 100%, and on who stands behind it, from 10% to 100%, matching hosted Jev on both. For hosted Jev, it turned "I can't tell" into an answer. It was right on every bare confirmation case, because each one called for "I can't tell", but only the files with MX got a yes or a no.

Apple's model was the one surprise. MX cut its made-up answers, but its accuracy fell in two places. One was a page whose metadata said it had been replaced, while the text still gave the old fact. Apple's model believed the text, and its score fell from 83.3% to 75%. The other was naming who stands behind a PDF, where it fell from 100% to 50%, and I don't yet know why. Some of its new mistakes were "I can't tell" where the answer was known. That's wrong, but it isn't invented.

These sets are small, and the cases are mine, written to test exactly this. Read the percentages as indicative, not as a figure for how much MX reduces hallucination on the open web.

Can a Rebuild Borrow Jev's Honesty?

If hosted Jev is honest and my rebuilds are fast, could Jev check their work? On 2026-10-16 I asked it to, under a registration I wrote before the first call. Each time a rebuild decided, hosted Jev saw the same file and question with that answer attached, and said confirm, reject or "I don't know". Only the sure answers it confirmed would act.

It worked on the wrong answers. The 8B rebuild was sure of a wrong answer 42 times; Jev let one of them through, and then not confidently enough to act. Of the sure answers it confirmed, none were wrong: 105 for the 8B and 127 for the 70B, against 1 in 6 and 1 in 10 unchecked.

It cost the right ones, though. Jev rejected about one right answer in five. Of the sure answers the rebuilds had got right, about half no longer acted: 105 of 206 still did for the 8B, 127 of 301 for the 70B. Hosted Jev deciding alone was sure 267 times and wrong twice. The checker also sees everything the rebuild saw, so the pair is no more private than calling Jev directly. If you can call hosted Jev at all, let it decide. If you can't, this check isn't open to you either, and a rebuild's sure answer is still a suggestion.

The Answer

MX first, then buy or build. MX isn't a third choice beside the other two, because both need it. Without it, a small decider guesses and a hosted one tells you it doesn't know, so either way the work comes back to a person.

Buy when the data is allowed to leave the machine and the decision acts on its own: routing a ticket, letting a command run, filing a document. That's where being sure and right matters, and hosted Jev is the only option I measured that holds up there.

Build when nothing may leave the machine, or when you want to understand what you're buying. It takes an afternoon, it's free to run, and it's quick. MX made my rebuilds more accurate and less inclined to invent, but I haven't shown it makes their confidence trustworthy. So treat a sure answer as a suggestion. Let it sort the easy cases and send the rest to a person or a bigger model.

Either way, keep a set of your own questions with known answers, and run every option against it before you trust one. Several of these results weren't what I expected, Apple's most of all.

How to Use Jev and MX

1. Put MX in the file. Declare the facts a reader needs as labelled fields that travel with it: status, dates, an expiry or review date, what replaces it, and who owns it, plus a short summary. Use frontmatter in markdown and the XMP packet in a PDF. The fields matter more than the prose.
2. Ask a bounded question. Give the decider a fixed set of answers, including an escape such as "cannot be determined". It returns one of them and a confidence, never free text.
3. Act only at the bar. At 0.9 or above, let the answer act on its own. Below it, pass the case to a person or a bigger model.
4. Choose where it runs. If the data may leave the machine, call hosted Jev. If it can't, run a local rebuild and treat its sure answers as suggestions. Don't have hosted Jev check a local one; let it decide instead.
5. Measure first. Keep a small set of your own questions with known answers, and run every option against it before you trust one.

MX makes the answer readable from the file. Jev makes the decision cheap and tells you how sure it is.

What I Take From It

The decision-model pitch is right about what it pitches. Splitting the decision from the writing makes it fast enough to run inside a loop and too cheap to count. I got that on a laptop with sixty lines.

What the pitch leaves out is that a decider is two declarations and a model. The first declaration is the answer set, and a small model can be handed an escape option and still not use it. The second is the state, and a model can be confidently consistent about a page that never says when it was published. Neither failure shows in its confidence figure, which only describes the choices and the state you supplied.

The model is the part you can buy. The state is the part you have to write. Put the declared fields in the file, where a script can find them, and every model that had been inventing answers invented fewer. All but Apple's got more right. The decider made each decision cheap, and the embedded metadata got more of them right. That's MX's job, and these are the numbers for what it's worth.

I've kept the evals, the runner, the run files and the script that rendered these tables. If you want to run them on your own models, get in touch and tell me what moved.

A declared interest. CogNovaMX sells the discipline these results support. What Jev is stands on TypeSafe's own published material, and the hosted figures come from my own calls to its API. I have no relationship with the company beyond using that API.

Related

• A Decider Is Only as Good as Its State: what a decision model is, and why its calibration stops at the state it's handed.
• Nobody Got Fired for Using GEO: why cheap decisions multiply, and why MX is worth promoting before the market asks.
• Reduce Inference, Save Energy: the Jevons paradox applied to the energy AI inference uses.
• Thought Is Getting Cheaper Faster Than Transistors Did: Epoch AI's price-of-thought figures set beside Moore's law and our own local decider.
• Humans Take Journeys, Agents Take One Fetch: why most agents read one file in isolation and decide, and why MX designs for both a drip feed and everything at once.
• Why Use MX: why a file needs metadata that travels with it, so the machine that meets it next doesn't have to guess.

Where Your Own Content Stands

These tests read each file on its own, the way most agents do. Whether your files declare enough to be read that way is what an AX (agent experience) audit measures.

• Get in touch about an AX audit
• The agent web series