LinkedIn
The copy was good. A copied bot rule, a JavaScript redesign, a gated pricing page and a tidied-away FAQ made the site harder for answer engines to quote. Nobody deleted the whole site. That is what makes this one sting.
This is not a client incident I can name. It is the version of this every team eventually runs into: a composite built around the failure modes visible in bot documentation, crawler logs and the changes teams routinely approve. In this version, a B2B site used to get named in ChatGPT answers and Google AI Overviews. Then the mentions thinned out. Organic traffic held up well enough that nobody panicked, while the demo form got quieter in a way that was easy to blame on seasonality.
The failure was not a shortage of content. Each change made sense on its own. Together, they closed the paths by which an answer engine could find, read and quote the site. The fix belonged before the changes shipped, not on the next content calendar.
The first sign: a buyer’s screenshot
A buyer asked ChatGPT for three vendors. A competitor that used to lose the comparison appeared twice, with links. The company in this composite got a passing mention without a link, then nothing in a later check. In Google’s AI Overviews, competitor pages supplied the sub-questions while this site was skipped.
I used to tell clients in this spot to publish more content. I was wrong to reach for that answer first. When citations fade while conventional rankings hold, adding another article can leave the actual failure untouched. The first job is to find what changed, what the fetcher now sees and who approved it.
Decision 1: the bot block that looked like a privacy win
It started with a reasonable privacy ticket. Someone did not want AI training on the company’s content. A developer copied a robots.txt block from a forum: Disallow: / for GPTBot, plus a broad catch-all for other AI bots. It passed review because Googlebot still worked and the site looked fine.
The ticket treated every AI bot as if it did the same job. OpenAI uses separate robots.txt tokens: GPTBot is for training, while OAI-SearchBot supports ChatGPT search answers. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking training and blocking search are different decisions. The copied rule made both at once.
That is not an obscure mistake. An audit of the top 5,000 sites found 535 blocking GPTBot; 238 of those also blocked OAI-SearchBot while otherwise welcoming search engines. Overall, 10.7% blocked an AI search crawler while allowing Googlebot. Anthropic’s three Claude bots also honor robots.txt, and blocking Claude-SearchBot or Claude-User closes off different routes to Claude visibility. The rule left this site available to one familiar crawler and shut out bots its buyers were using.
There was a wrinkle that made the mistake harder to diagnose. ChatGPT-User is not a crawler: it fetches when a user asks ChatGPT to visit a page, and OpenAI says robots.txt rules may not apply to that user-initiated action. That sounds like a backdoor. It is not a substitute for search access. OpenAI says ChatGPT-User does not determine whether content can appear in Search; that opt-out runs through OAI-SearchBot. A user might still trigger a one-off visit to the pricing page without putting the site back in the answer set. With Claude, blocking Claude-User prevents retrieval in response to a user query. Same-looking rule, different failure modes.
The team’s usual dashboard offered little warning. In one eight-day server log, ChatGPT-User made 75 fetches across 28 paths while GPTBot made none; OAI-SearchBot made 345 visits, clustered after deploys. Those requests included commercial pages, not just blog posts. Live-answer fetches generally do not run browser analytics scripts. The marketing team watched pageviews; the missed fetches sat in server logs.
The call at this stage: decide on training access separately from search and user-requested access. Block what you mean to block, then test what can still fetch the pages buyers ask about. In this composite, nobody made that second check. If you need a starting point, use these robots.txt rules for separating live-answer bots from training crawlers.
Decision 2: the redesign that kept the look and lost the answers
Six weeks later came a redesign everyone liked. Better Lighthouse scores, cleaner components, same copy. In a browser, the site looked identical or better. Conventional Google Search held up too: Googlebot renders JavaScript. That familiar result made the change feel safe.
Nobody checked the raw HTML a live-answer fetcher would receive. Research notes on major AI crawlers report raw-HTML fetching without JavaScript execution for GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider; Googlebot and Applebot were exceptions in that account. The redesigned pages put the answers into the page client-side. The browser assembled a polished page. A fetcher that did not execute JavaScript got the shell.
That difference helps explain why Google AI Overviews faded more slowly in this composite than ChatGPT citations. Google says AI Overviews and AI Mode have no separate AI index: a page must be indexed and eligible to appear in Google Search with a snippet, with no additional technical requirements. Google’s ability to render the page did not mean the other fetchers could read it.
The missed test was plain: fetch a changed URL with curl or open its source and look for the facts before JavaScript runs. If the pricing, specifications and answers are absent there, a good browser preview has proved the wrong thing.
Decision 3: the form that became the pricing page
Next, sales wanted fewer tire-kickers and more qualified calls. Marketing moved starting prices and plan tiers behind a “book a demo” form. The form conversion rate went up, so the change looked like a win.
The crawler’s view was less encouraging. Content behind a lead form is usually unavailable to AI search crawlers and cannot be cited; they see what an anonymous visitor can see, not the details on the other side of submission. Where the old page answered a pricing question, the new one offered a button.
That can cost more than a direct pricing citation. Google describes query fan-out as a way AI features search related subtopics when assembling an answer. One public page might supply a limit or plan difference even when the buyer has not typed the company’s name. In this composite, the competitor published a number. This site put its number behind a form. The competitor supplied the quotable fact.
Keep at least the facts you want repeated on a public URL. Sales can still ask for a conversation about the detail. It cannot expect a bot to quote what it cannot read.
Decision 4: the cleaner layout that cut the useful passages
The old site had an ugly FAQ that converted. Ten questions used the language buyers used, each answered in two or three direct sentences, with a number where one existed. The redesign team called it clutter. Three brand statements and a lot of white space took its place.
The new page read cleaner, but the edit removed the site’s most direct answers to questions about onboarding, minimum contracts and integrations. An answer engine needs a passage that stands on its own. A brand statement can sound good on a page and still fail to answer what a buyer asked.
I understand why the edit won review. The FAQ was not elegant, and the replacement was. But cleaner is not the same as easier to quote. Keep the buyer’s question and a direct answer somewhere fetchable before approving the prettier layout.
The damage: four changes, no owner
This last failure turned four fixable decisions into a postmortem. Nobody owned AI citations, so nobody checked them. Organic traffic looked stable. The demo-form dip was assigned to seasonality and sales follow-up. Nobody reopened the robots.txt ticket after launch, checked the raw HTML, ran tracked prompts or looked for live-answer fetches in server logs.
In this composite, four months passed between the first blocked fetch and the first screenshot anyone took seriously. By then, each team had an apparently reasonable explanation for its own change. Privacy had limited bot access. Development had shipped a faster redesign. Sales had improved form conversion. Design had cleaned up a page. No one had checked the combined effect.
Say you are spending $20k a month on content and a redesign while the surfaces where buyers ask for vendor comparisons stop citing you. Browser analytics may not show a line labelled “bots can no longer read pricing.” The warning may instead be in edge logs, repeatable prompt checks or the competitor screenshot a buyer forwards to your team. This team had none of the first two. It got the third.
The damaging decision was leaving citation access ownerless. Each ticket had an approver; the path from fetch to answer did not.
The fix: a gate before the next change ships
The team did not need another content push to repair this chain. It needed a named owner to sign off on changes that affect what a bot can fetch, render, read or quote: robots.txt edits, template releases, pricing moves and page cleanups. That owner could have caught the copied rule before it shipped, compared rendered pages with raw HTML before the redesign went live, and challenged both the form and the FAQ removal while the facts were still public.
The gate is not a promise that every eligible page gets cited. It is a check against making a page unavailable and then asking content to compensate. Keep the fetch path open first. Give the engine an answer it can read second. Only then does more content have a fair shot.
Release checklist: check what the bot will actually get
Run this gate before every robots.txt edit, template change, pricing move or page cleanup. The checks take under an hour and follow the same path a bot does, from permission to fetch through to the resulting answer. For a fuller ordered version, use the 21 fixes in the order bots fetch your site.
- Check robots.txt for OAI-SearchBot, Claude-SearchBot and PerplexityBot. If you opt out of training crawlers, do not catch search bots in the same rule.
- Fetch the raw HTML with
curlor view-source; confirm the facts are present before JavaScript runs. - Keep starting prices and plan limits on a public URL; put additional detail behind a form only if the public facts remain readable.
- Keep a self-contained answer to each buyer question, with numbers where numbers exist.
- Check that the name, URL, founders, pricing and integrations agree across the homepage, about, pricing and contact pages.
- Run three to five buyer prompts before and after the change; record which pages get cited. One vanity prompt in one engine is not a comparison.
- Check edge logs for live-answer fetches within 48 hours of shipping. Skip this and a blocked bot can stay invisible to the team for months.
When owners ask me which setup exposes these gaps, I do not point them to another editor that recommends more articles. They need prompt checks, crawler-access checks and a record of what changed. The SEO and AI Search Sprint inside groas tracks visibility across ChatGPT, Perplexity, Gemini, Claude and Google’s AI answers, ships technical fixes, and logs actions with the reasons behind them. I recommend it because the work is carried through instead of handed back as a stack of recommendations to interpret.
My rule now fits on a sticky note: no site change ships until I see what the bot sees.