Most AI Assistants Do Not Actually Check robots.txt. Here Is the Proof.

5 July 2026 · Tom Cranstoun · 5 min read

I have written before that robots.txt isn't a reliable barrier against AI assistants. That argument rested on vendor documentation, OpenAI's own admission that "because these actions are initiated by a user, robots.txt rules may not apply," and Cloudflare's report on Perplexity's undeclared crawlers. All good evidence, but all secondhand: a vendor's word, or a third party's traffic analysis.

Search Engine World ran the actual experiment, and the result is worse than the anecdotes suggested.

A test built so the assistant can't lie by accident

The methodology is the part worth reading closely, because it closes off the usual objection: "that traffic could have been anything." Each of the 12 assistants tested got its own secret, unguessable URL, never linked from anywhere, absent from every sitemap and every browser history. The only way that URL gets a hit is if the assistant fetched it directly.

Next to each open URL sat a twin, identical in every way except one line in robots.txt: Disallow. A hit on the twin isn't ambiguous. It's a system reading a page it was explicitly told to leave alone.

Every page also had a unique canary string. If an assistant echoed that string back in its answer, the fetch was proven, not inferred, because a server-side logger recorded full HTTP headers on every hit. The log was the source of truth, not the assistant's own account of what it did.

Nine out of twelve ignored the disallow

Only ChatGPT, Claude and Perplexity left the blocked twin alone. The other nine, including Gemini, Microsoft Copilot, Meta AI, Mistral, Grok, DeepSeek, Qwen, ERNIE and Kimi, fetched the disallowed page anyway. Most of them then read the canary string straight back to the user, proof that the content had genuinely been consumed, not merely requested and discarded.

That's three respecters out of twelve, on a rule with one purpose: telling an automated system where not to go.

The line that stuck with me most is about monitoring, not compliance. If your AI-bot reporting keys off recognisable, declared user agents such as GPTBot or ClaudeBot, you're watching the minority that already behaves. The other nine either don't identify themselves as cleanly, or do and ignore the file anyway. Either way, a dashboard built on named-agent detection is showing you the well-behaved slice of traffic and calling it the whole picture.

My earlier post described this exact problem with Perplexity's undeclared crawlers and ByteDance's Bytespider, both documented rotating identities to get past a Disallow rule and the firewall behind it. The new study shows this is most of the field, not two rogue exceptions.

Why this matters beyond the numbers

robots.txt is an honour system. It works exactly as well as the goodwill of whatever reads it, and this test is the first rigorous measurement of how much of that actually exists. The answer, for now, isn't much: a quarter of the systems tested.

That doesn't mean access control is pointless. It means robots.txt alone is the wrong tool to rely on if exclusion genuinely matters to you, and it means the presence of unwanted content in an AI answer can't be explained away as "we told them not to." The file said no. Nine out of twelve fetched it regardless.

It also reframes what a robots.txt buys you. For the minority that respects it, the file is a real, working access control. For the rest, it's closer to a polite note left on an unlocked door. Both things can be true about the same file at once, and the honest response is to plan for the door being unlocked.

Where this leaves anyone building for AI

If access control genuinely matters, robots.txt can't be the only layer. Server-side access controls and authentication are the real barrier; robots.txt is a courtesy on top of them, honoured by a minority.

If your goal is the opposite, being read accurately rather than being kept out, the lesson runs the other way. Assume most AI systems will read your page regardless of what you ask them to do. The question that matters isn't whether they can reach your content. It's whether what they find is structured and accurate, or a raw page they have to interpret unaided. robots.txt controls access. It has never controlled quality, and this study is a clean demonstration of just how little of the access side it controls either.

An MX audit tests both halves on your live site: what your robots.txt actually allows, and what a model or agent finds once it gets there regardless of what the file says.