The AI Bot Traffic Fallacy | Tom Cranstoun

Hosting providers are sounding alarms about AI crawlers. GPTBot, ClaudeBot, Bytespider and the rest now account for a serious share of what some hosts serve, and the growth is steep. Cloudflare's figures for the twelve months to May 2025 showed crawler traffic up by nearly a fifth, with GPTBot alone more than quadrupling. The prescribed remedy is always the same. Tighten the WAF rules. Rate-limit by user agent. Write a more aggressive robots.txt. Treat the crawlers the way you'd treat a DDoS attack.

I understand the instinct. The bandwidth bill is real, so is the server load, and for a site that never asked to become training data, so is the resentment. But treating AI bot traffic as a security threat is a category error.

The surge isn't an attack. It's friction: the sound of an obsolete web architecture serving autonomous machines with pages built exclusively for human eyeballs.

The DOM Bloat Tax

Think about what an agent actually does when it visits a typical page to answer a question.

It wants a fact. The opening hours, the price, the spec, the date. Fifty bytes of information, maybe less. To get those fifty bytes it downloads megabytes of DOM, and if it's one of the rare fetchers that renders JavaScript, it executes your framework bundles too. Then it burns inference tokens stripping out the navigation header, the cookie banner, the newsletter modal, and the marketing boilerplate, just to find the sentence it needs.

Every layer of that page exists for a human. The hero image, the animation, the carefully staged scroll experience: all of it built for eyes, none of it useful to a machine, all of it on your bill every automated visit.

Blaming the bot for the resource drain is like blaming a train for damaging a dirt road. The train is doing nothing wrong. The road was never built for it. And the answer to trains was never "ban trains"; it was rails.

The Invisible Web Trap

The blocking remedy has a second problem, and it's the bigger one: it works.

Block the crawlers at the perimeter and they stop coming. Your bandwidth bill drops. And your content quietly disappears from the layer of the web where a growing share of discovery now happens. Assistants answer questions about your market from your competitors' pages. Agents comparing suppliers never see your offer. The models that people increasingly consult first have nothing current about you to say.

I wrote about the mechanics of this in my robots.txt piece: blocking a retrieval crawler removes you from that product's answers, and each block is its own decision, with a cost. The perimeter defence doesn't protect your content. It erases you from the agentic web while the sites that stayed readable take your visibility.

That's commercial self-harm with a clean bandwidth graph.

Bifurcated Delivery: UX for Humans, MX for Machines

The way out isn't a better firewall. It's recognising that you now serve two audiences, and only one of them wants your DOM.

UX asks: can a human understand and navigate this? MX, Machine Experience, asks: can a machine understand and act on this? A mature architecture answers both. Humans get the designed page. Machines get a structured, declared payload of the facts, the metadata, and the provenance.

The difference in cost isn't subtle. Serve an agent just the fifty bytes it wants, plus the metadata saying where they originated and how long to trust them. No script execution required. No boilerplate stripping. No inference spent guessing which sentence is the answer. The agent gets a better result, and your infrastructure still serves it.

And once the machine channel exists, you can go further than serving requests cheaply. You can stop forcing machines to re-ask. A crawler re-fetches your page on a schedule because it has no way of knowing whether anything changed. Declare your content's freshness and state transitions explicitly, and the re-crawl becomes a subscription: the machine checks a lightweight declaration, sees nothing has changed, and leaves. The most efficient request is the one that never needed to happen.

Governance Over Firewalls

A WAF rule can only say yes or no. That's the simple vocabulary. And as the stealth crawler episodes show, a determined operator treats "no" as a routing suggestion anyway.

A machine contract can say something useful: here is the structured version, here is what it costs, here is the attribution I require, here is how fresh it is, here is the proof of who published it. Access terms a machine can read and act on, declared in metadata rather than enforced by guesswork. Rate limits tied to identity instead of user-agent strings that anyone can fake. Attribution requirements stated where the agent can actually see them. Licensing terms a machine can honour at the moment of use.

Governance beats defence because governance scales with good actors, and they're most of the traffic. The honest crawlers already identify themselves, already respect robots.txt, already publish their IP ranges for verification. They'll happily take a cheaper, cleaner, declared payload. Give them one, and your firewall budget can concentrate on the small population that actually behaves like an attack.

Stop Fighting the Machines

The hosting providers are right that something has to change. They're wrong about what. The web doesn't need higher walls; it needs a machine lane. The traffic they're trying to repel is the demand signal for it.

Machine traffic isn't going back down. The organisations that treat it as an audience worth serving on declared terms will stay visible in the agentic economy, cited by assistants and open for agent business. The ones that treat it as a siege will win the bandwidth battle and lose the visibility war.

Stop fighting the machines. Start governing their experience.

If you want to know what agents currently see when they visit you, and what it's costing both sides, an MX audit tests your live site the way the machines read it and reports back in a form you can verify yourself.