The Cheapest Bot Is an Honest One

Most of what visits your website now isn't a person. A large and growing part of your traffic is machines, and the direction is only one way. Every one of those visits costs you something real: origin load, bandwidth, the compute behind a render, the crawl budget you were saving for the pages that matter. The bill is machine-shaped, and it's growing.

The usual response is a wall. Fingerprint the traffic, score it, and block whatever looks automated. That works, and it's also where I want to push back.

The wall is aimed at the wrong thing

Start with which machines are actually expensive.

The crawlers that identify themselves are the easy ones. A search engine's crawler, one of the named AI crawlers, a partner's integration: they announce themselves, they behave, they respect a rate limit, and you can plan for them. Predictable machines are cheap to serve.

The expensive visits are the ones that hide. A scraper wearing a copied browser user agent. A monitoring script pretending to be a person. You pay for these not because they're clever but because you can't tell them apart from a real visitor, so you hand them the whole heavy page each time they call.

So the cost isn't that machines visit; it's that some of them won't say what they are. A firewall rule treats the whole machine web as the threat, then spends forever trying to guess which visitor is lying. Scrapers rotate their addresses and copy fresh fingerprints the week after you block the old ones. It's a treadmill, and you don't get off it by running faster.

First, stop paying to be slow to yourself

Before blaming anyone outside, look at what your own stack is doing. A surprising amount of machine cost is self-inflicted:

• Dead pages answered with a success code instead of a "not found", so every broken-link probe and scraper miss counts as a live page served.
• Uptime checks that fetch a whole page every minute when a lightweight header request would answer the same question.
• A proxy somewhere in the chain that strips the original user agent, so the honest crawlers you meant to wave through arrive looking anonymous, and get served the expensive way after all.

That isn't a bot problem. It's plumbing, and fixing it costs nothing but attention. Do this first. It's the least glamorous part of the work and often the largest number on the page.

Then make honesty the easy path

Here's the move I'd make instead of building the wall higher.

You can't tell an honest agent from a scraper by guessing harder. You can tell them apart by giving the honest one a cheaper, better path it'll actually take, and letting your rate limiter handle whoever refuses it.

Two parts to that.

Serve machines the cheap version. An agent that wants your content doesn't need your fonts, your analytics, your consent banner, or the JavaScript that assembles the page inside a browser. It needs the content. Give it a light, structured representation, clean markdown or a JSON view or a description file it can read directly, and a well-behaved agent will take that instead of rendering the heavy page. That's cheaper for you and better for the agent at once. Nobody loses. And the scraper that insists on rendering the whole thing anyway is now the obvious outlier, easy to slow down without touching anyone real.

Price honesty, don't gate it. You're not banning the anonymous traffic. You're giving the identified traffic the fast lane: the clean structured payload, a fresher cache, a little more room on the rate limit. Whatever won't name itself gets the slow lane. You never have to decide who to block. You make declaring yourself the path of least resistance, and let the rest queue.

Legible and cheap are the same property

The thing worth seeing is that these aren't two projects.

The structured version that lets a machine read your content is the very thing that makes you cheap to serve it. An agent that proves it's well-behaved hands you the same signal that lets you shape traffic without a fight. Readable by a machine and cheap to serve it are one property, seen from two sides.

This is most of what we mean by machine experience: a page, a document, a feed that says plainly what it holds and how to use it. The payoff people reach for first is that agents can then understand you, cite you correctly, act on your content, send you the customer. The quieter payoff is the one on the invoice. A machine that reads you in a single small fetch is the cheap kind. The machine that announces itself and its purpose is the one you can safely wave through.

The honest caveat

This is a slower fix than a firewall rule, and I won't pretend otherwise. A rule at the edge cuts your bill straight away. The structured route cuts it for good, but only once you've built it and meant it, which is real work.

There's a reason to prefer it beyond cost. The machine web isn't going to get smaller. Today's anonymous scraper is sometimes tomorrow's paying customer's shopping agent, sent to buy from you. A strategy whose only tool is "block what I don't recognise" is at war with its own future traffic. Better to build the front door than to keep raising the wall.

None of this is only theory. You can watch the honest-and-hiding split get measured in the open. One independent experiment runs honeypot paths blocked in its robots.txt and publishes, live, the crawlers that respect the rule and the ones that trip the trap. The column worth watching verifies a crawler's cryptographic signature, and that's the part that answers an objection I stepped past. A copied user agent buys anonymity; a signed request doesn't, and the honest crawlers are starting to sign. Once an agent can prove its identity rather than just assert it, a scraper can't fake its way into the fast lane.

You can spend the next few years getting better at spotting the liars. Or you can make honesty the easy path, serve the honest ones cheaply, and keep the wall for whoever still won't knock.