The Structured Index Buys Precision, Not Reach

Somebody finally ran the experiment the GEO argument keeps assuming.

A Google team built two agents with the same architecture and pointed them at two different corpora. One queried the Google Search index. The other queried Google Dataset Search, a curated corpus of 90 million records where inclusion requires schema.org/Dataset or DCAT markup. Then they measured what each agent brought back.

The structured agent was much more precise. It also answered 29% fewer questions. Both halves of that sentence are in the paper, and only the first half is being quoted.

What They Actually Did

Shiyu Chen, Tarfah Alrashed, Alon Halevy and Natasha Noy, all at Google in Mountain View, posted Do Data Agents Need Semantic Metadata? to arXiv in May 2026, revised in July. It's a preprint. There's no journal reference and no venue named on it, so treat it as unreviewed work by the people who built the system under test.

The query set is 58 English keyword questions from the Information Retrieval subtask of the NTCIR-16 Data Search 2 benchmark. Real crowdsourced phrasings, short and underspecified, of the "air quality baltimore" kind.

The controls are tight. Same framework, same model, temperature zero. Three results per query, the same JSON shape out, and a fixed "no relevant datasets found" string both agents had to return rather than improvise when they came up empty. An LLM-as-a-judge pipeline then scored every returned URL on relevance, accessibility and page type, with "fully FAIR-compliant" reserved for a result that scored top marks on all three at once.

What the Paper Proves

That's a real effect, and the precision figure carries statistical significance. The structured agent was much better at coming back with something you could actually open.

The mechanism the authors give for those open-web failures is worth reading closely, because it isn't about ranking. It's about what happens after retrieval: page markup causes token bloat, dense prose degrades the context window, and portal pages trap an agent in redundant search loops instead of handing it a payload. They call it a last-mile utility failure. The open-web agent found relevant pages perfectly well. It just couldn't get the data out of them.

The Two Numbers the Summaries Drop

Relevance didn't move. Rated highly relevant: 60.4% for the open-web agent, 60.7% for the structured one. Statistically that's the same number. Markup didn't help the agent find data that matched the question better. It helped the agent find data it could open. Those are different products, and only one of them is what most people think they're buying.

Coverage collapsed. The open-web agent answered 56 of the 58 queries. The structured agent answered 40. The authors attribute the gap directly to limited schema.org adoption, and say it's most pronounced in the long tail, in the research domains and repositories where dataset pages carry no annotation at all.

So here's the honest summary of the trade. Inside the curated index your odds improve a lot. Outside it you don't rank badly, you're absent by construction. And the agent, by the paper's own framing, is built to prefer an empty result over a probabilistic guess.

Three Things the Confident Version Gets Wrong

I've seen this study written up as "only the index changed". The paper declines that framing itself.

The ranking differs too, and they say so. In the limitations the authors write that Google Search and Google Dataset Search share core infrastructure but that their internal ranking mechanisms remain black boxes, and that rather than isolating the algorithmic delta they're measuring end-to-end utility. So this isn't a clean single-variable experiment. It's a comparison of two ecosystems as they actually behave, which is a more useful thing to measure and a weaker thing to claim.

The query handling differed by design. The open-web agent appends data-seeking keywords such as "dataset" to the user's intent, because without that expansion it'd be a straw baseline. The structured agent skips the expansion, because the vertical already restricts results to dataset entities. That's defensible and it's documented, and it's also a second difference between the two arms.

It's a small study, and the authors flag that first. 58 queries, 164 URLs against 112. They write plainly that the query set may not capture the diversity of the long tail, and that the result-density difference between the two agents wasn't statistically significant at all. Only the URL-level precision figure clears p<0.01. Anyone quoting a second significant finding out of this paper is inventing it.

The Filter Nobody Mentions

Buried in the method section: Google runs a classification model over candidate pages and filters out roughly 80% of them for invalid or misused annotations before they reach the 90 million that remain.

Read that again if you're about to bolt Dataset markup onto a blog post with no downloadable file behind it. That's precisely what the filter exists to catch. Marking up a page as something it isn't doesn't get you into the clean index. It gets you classified as noise, by a model built for the purpose, and you never find out.

It's the same failure I keep seeing from automated markup: an agent told to "add structured data" can't tell a fact that carries weight from a decorative one, so it tags everything. Less, chosen well, beats more.

Their Own Recommendation Isn't "Add Markup"

The paper proposes a hybrid architecture, and it proposes it for the agent builder rather than the publisher. Query the high-precision semantic layer first. If it returns nothing, fall back to unstructured retrieval and cast a wider net. The authors are explicit that unstructured retrieval remains the right tool for broad exploratory work, and that the structured corpus trades away recall for reliability.

That's a considerably more modest conclusion than "structured data wins", and it's the one their numbers support.

Where I Go Beyond the Paper

I want to be careful here, because the study is about a dataset search vertical feeding an agent. It says nothing about how ChatGPT, Perplexity or AI Overviews pick citations. Another corpus, another task, another scoring rubric.

My working assumption is that the retrieval half generalises, and the paper gives a reason to think so: the failures it describes aren't specific to datasets. Token bloat from markup noise, prose that degrades the context window, portals that loop an agent instead of answering it. Any agent parsing any page hits those. An agent that has to reconstruct a number out of prose will often return nothing rather than risk returning the wrong one, and returning nothing is invisible to you.

What doesn't carry across is citation behaviour. Whether a model puts your URL under its sentence is a ranking and attribution question, and this paper measures neither. So: the failure mode generalises, the citation outcome is untested. If someone tells you this study proves schema.org gets you cited, they haven't read past the abstract.

What It Means If You Publish Data

You're not making your page more citable. You're choosing to compete inside a smaller, cleaner index instead of a larger, noisier one.

Inside it, your odds improve. Outside it, you're not there at all. That's a positioning decision about which room you want to be judged in, and it deserves a conversation about who your data is for, not a ticket in a sprint labelled "add schema".

Two questions settle it faster than any audit. Is there a machine-readable file at the end of it, or only a page describing one? And is the audience you care about reaching you through a curated vertical, or through open-web search? If the answer to the first is "only a page", the markup is a liability rather than an investment.

Where MX Fits

The result I keep coming back to is the flat one. Relevance didn't move. Structured markup didn't make the data a better answer to the question. It made the data openable. Findability was never the bottleneck. The last mile was.

That's the MX argument arriving from an unexpected direction. Making a page machine-readable isn't a ranking tactic, and the paper's numbers say so plainly. It's what stops a machine having to guess at the moment it tries to act on what it found. The agents in this study didn't fail because they couldn't find the right page. They failed because, having found it, they couldn't get anything out of it.

There's a layer past even that one, which this study doesn't touch because that wasn't its question. A curated index tells an agent that a record met an inclusion rule. It doesn't tell the agent who published the underlying file, when, under what version, or whether anything has changed since. Google's classification filter is a quality gate, not a provenance record. When a file travels beyond the index that catalogued it, and files always do, what it carries on its own is all that's left. Readable, so the agent can use it. Attributable, so the agent knows who stands behind it. That second half is the part no vertical index supplies, and it's why I wrote about the provenance layer schema.org still doesn't have.

If you want to know which of your own pages an agent would find and then fail to use, an AX (agent experience) audit fetches them the way these systems do and shows you the text that actually comes back.

The longer argument about why none of this is a ranking trick is in GEO is a tactic, MX is the specification, and the companion piece on what Google's own guidance does and doesn't endorse is Google Named GEO, Then Debunked Most of It.

Reference

Chen, S., Alrashed, T., Halevy, A., & Noy, N. (2026). Do data agents need semantic metadata? A comparative study in agentic data retrieval (arXiv:2605.28787v2). arXiv. https://arxiv.org/abs/2605.28787