Training vs Inference.

Every AI model that mentions your business got its information one of two ways. Either it absorbed your content months ago, during training — or it is fetching your page right now, while a user waits for an answer. These are fundamentally different mechanisms with different consequences, and understanding the difference is the single most important insight in Machine Experience.

Large language models are trained on massive datasets — Common Crawl snapshots, Wikipedia dumps, curated web archives. This happens months or years before the model ever processes a user query. Your website content, as it existed when the training data was collected, gets baked into the model's weights.

• Historical, not current. A model trained on your 2024 site will confidently describe your 2024 offerings even if everything has changed since.
• Comprehensive but frozen. Training crawlers index broadly across your site, but once data enters a training set there is no mechanism to update or remove it.
• Generally respects robots.txt. Common Crawl honours exclusion rules, though compliance is inconsistent across all training data sources.
• Stale impressions compound. If your company restructured, rebranded, or changed pricing since the training data capture, the model carries outdated impressions indefinitely.

The consequence: wrong guidance in recommendations, delivered with confidence, over months or years.

This is the newer, more consequential access pattern. When a user asks "What does company X charge for their premium plan?", the machine fetches your website right now, during the conversation. This is inference-time access, and it is growing fast — AI referral traffic to retail is surging year-over-year.

• Real-time. The machine reads your current page, not a historical snapshot. What it finds (or fails to find) determines its answer immediately.
• Selective. Unlike training crawlers, inference-time agents fetch specific pages relevant to the user's query, not your entire site.
• May not respect robots.txt. There is no industry consensus on whether inference-time fetches should honour traditional crawl restrictions.
• Immediate consequences. If your page fails for a machine at inference time, you lose that customer right now. There is no second chance.

The consequence: a lost transaction you never knew about, because the machine could not read your page.

Aspect	Training-time	Inference-time
When	Months or years before use	During the user's conversation
Source	Historical snapshots (Common Crawl)	Current live pages
Scope	Comprehensive crawl of entire site	Selective — specific pages only
Updates	Frozen in model weights	Always current
robots.txt	Generally respected	No consensus
Business impact	Stale impressions over months	Lost customer right now

Inference-time access is where MX patterns pay off most directly. When a machine fetches your page during a live conversation, every structural failure becomes an immediate business failure:

• If pricing is in a JavaScript-rendered widget with no server-side fallback, the raw parser sees nothing — and tells the user it could not find your prices
• If product comparison data lives inside a canvas chart, the machine cannot extract it — your competitor's plain HTML table wins
• If your page uses div elements instead of semantic HTML, the machine wastes tokens guessing at structure — and guesses wrong
• If your content hides behind a cookie wall or consent banner, the machine sees only the overlay — never the page beneath it
• If dynamic pricing changes between requests, the machine gets two different answers and no way to know which is current

The good news: you do not need different strategies for training-time and inference-time access. Proper semantic HTML, structured data, and explicit metadata improve both simultaneously.

• At training time — clean structure means better representation in the training data, producing more accurate baseline knowledge about your business.
• At inference time — the same clean structure means machines extracting current information do so accurately, without hallucination.

This is the core MX insight: every pattern that helps one access mechanism helps both. The difference is that inference-time makes the consequences immediate and visible.

The five-step action framework from MX: The Introduction applies directly here: Audit, Understand, Prioritise, Implement, Measure. Start by understanding what machines actually see when they fetch your pages — not what humans see in a browser.

For the full technical foundation, including the five types of machine that visit your site and how each reads your content differently, download the free Introduction or continue to MX: The Handbook for implementation patterns.