AI Visibility Starts at the Fetch: Where the Chain Breaks

Last month, 15,515 requests claiming to be AI bots hit our robots.txt. We could verify 4,047 as real crawlers. Before asking whether an assistant likes your content, you need to know whether its fetcher reached that content at all.

That is the plumbing problem behind a lot of AI visibility advice. A live fetch may need to pass your robots rules, get through your edge security, retrieve a usable page and find a passage it can quote. Fail early, and a better headline or a fuller FAQ changes nothing. I will follow that path in order, using those requests as the running example. Content quality comes after the fetch path works.

First, identify the bot doing the fetching

Everything else follows from one distinction: a training crawler and a live-answer fetcher have different jobs. OpenAI runs separate bots for training, search indexing and user-initiated fetching. GPTBot ingests content for training; OAI-SearchBot builds the ChatGPT search index; ChatGPT-User fetches pages in response to a person’s request. One name in an access log does not stand for all three jobs.

I used to tell clients to block GPTBot first on principle. I was wrong to treat that as an answer to every AI visibility question. If the goal is a citation when a buyer asks a question next Tuesday, blocking a training crawler does not diagnose whether a live-answer fetcher can read the page. You have to look at the bot involved in retrieval.

That distinction gives us a workable model. Say a buyer asks an assistant about a product you sell and your page could answer. The assistant’s path to that answer depends on what is allowed, what returns when it asks, what text appears in the response and whether that text contains a useful passage. Trace the job, not just the bot name.

Robots.txt is a permission signal, not a delivery receipt

A crawler checking permission requests robots.txt. In our measured traffic, 1,231 requests to that file came from live-answer fetchers rather than training crawlers. That tells me where to begin an investigation; it does not tell me that any of those fetchers reached a product page. A request for the rules is not a request for the answer.

The rules are not a perfect prediction of behavior, either. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the fetch. Anthropic says its bots, including Claude-User, honor robots.txt. The same file can therefore lead you to expect different behavior from different fetchers.

Start with the rule for the bot and page you care about. Then check the page request and response. An allow rule is permission, not evidence of a successful fetch; a disallow rule should not be mistaken for proof that every user-initiated fetch stayed away. Read the file, then verify the behavior.

Step 1: The edge can refuse a bot you allowed

Suppose robots.txt allows ChatGPT-User onto our example product page. The request still passes through whatever protects the site before it reaches the origin. A security challenge that a person can solve in a browser may stop a crawler cold. From the site owner’s desk, everything looks normal: the file says allow, and the page opens in Chrome. Neither observation proves the fetcher received it.

The configuration matters. Cloudflare Bot Fight Mode can challenge suspected bots and cannot be bypassed with a WAF Skip rule; Super Bot Fight Mode offers a Skip-rule route. AWS WAF CategoryAI can block verified AI bots unless you override it. Those are different controls, so a generic instruction to “whitelist the bot” is not enough. Find which control acted on the request before changing one.

This is where the running example’s large robots.txt count can mislead. Even a verified fetcher requesting that file has not necessarily cleared the edge on its next request. Check the edge decision for the page URL and the fetcher you intend to admit. If a challenge or block answers the request, rewriting the page is work performed behind a locked door. Check the edge before you edit the copy. If you run groas, this is the sort of fault its technical audit is built to catch.

Step 2: A permitted request still needs a usable response

Permission gets the fetcher to the door. Retrieval decides what comes back. For our example page, the useful result is a successful response containing its HTML. A redirect chain can delay arrival; a bot-only 403 can refuse it; a 429 can treat a burst of fetches as abuse; a 500 can leave it with nothing. I have seen all four on sites whose owners swore the site was up. It was up for them in Chrome. That was a different request.

A live-answer fetch happens on a short clock. The more detours or failures between the requested URL and the page body, the less useful the page is at answer time. Do not infer success from a homepage loading, a search ranking or a robots.txt hit. Follow the request for the specific page and inspect its status and body. A page that ranks on Google can still be skipped by ChatGPT when its live fetch fails.

The repair is deliberately dull: serve the final URL directly where you can, look for different responses to bots and browsers, and inspect the IP alongside the user agent when debugging. A user-agent label alone will not tell you who made the request; I will come back to that. For now, the gate is simple. If the fetcher does not receive the page body, it cannot use a passage from it.

Step 3: The response body may not contain the answer

Now assume the fetcher gets a successful response. What text is actually in that response? That question matters on pages that depend on JavaScript to put their main copy on screen. A person sees the completed page after the browser runs scripts. A crawler that reads the initial HTML may see the title and a mostly empty body.

A Vercel and MERJ test found that several major AI crawlers fetched JavaScript files without executing them. The draft’s running product page does not need to be broken for every visitor to be broken for one of those fetchers. It only needs to place its answer somewhere the fetcher never renders.

I use the crude check first: view source, not inspect, and read the page without relying on scripts. Is the answer there? Are the price and headings there? An audit that finds a title but does not read the body can miss the failure entirely. If the source contains an empty shell, improve what the crawler receives before debating whether the visible copy is persuasive. Put the answer text in the HTML the fetcher can read.

Step 4: Make the answer easy to lift without losing its meaning

Once the fetcher has readable text, the problem changes. It can encounter your page and still choose another passage. Retrieval systems can split candidate pages into chunks and select a self-contained passage that answers the prompt. A page that scatters its answer across an introduction, a comparison widget and a closing paragraph makes that job harder than it needs to be.

For the example product page, I would put the direct answer under a heading a buyer can recognize, then support it with the details below. If the page compares options, use a real table or list rather than burying the comparison in an image. If a price appears only in a script-injected widget, this is still a rendering fault, not a request for prettier prose. Keep the diagnosis tied to the step that failed.

I think of this as plumbing followed by packaging. The first three steps deliver readable material. This one makes a useful part of that material stand on its own. That does not mean every answer needs an identical 40-word paragraph, and it certainly does not guarantee a citation. It means the reader and the extractor should not have to assemble your point from four pieces. Lead with the answer; make the supporting detail earn its place.

Verify the crawler before you trust the counts

We have traced what should happen to a request. Now return to the 15,515 requests that prompted this exercise. A user-agent string is a claim, not an identity check: someone else can send a request labeled GPTBot. Verification uses the source IP against the vendor’s published ranges, with DNS checks where appropriate. A two-week analysis found spoofed requests among traffic claiming known AI crawler identities. That is why I do not treat a bot name in a log as proof of demand.

Only 4,047 of our robots.txt requests passed verification. The remaining 11,468 are unverified requests, not 11,468 proven spoofs. That difference matters. If I call every claimed bot a real crawler, I exaggerate the activity I am trying to understand. If I call every unverified request malicious, I claim more than the check establishes. Neither helps me answer the practical question: did a verified live fetcher request the page, and what did it receive?

Keep those questions separate in the logs. Count verified traffic before drawing conclusions about crawler behavior. Then follow the relevant requests through permission, edge decision, status and response body. The gap between 15,515 and 4,047 is a warning against using a headline count as a diagnosis. Verify the actor before diagnosing its journey.

What a successful fetch still cannot promise

A page can pass every gate above and still go uncited. The assistant may have other candidate pages, and their passages may answer the question more directly. Freshness and corroboration can matter too. The fetch path explains whether your material gets a fair chance to be considered; it does not explain every choice made after that.

That boundary keeps the work in order. If our example page receives a verified live fetch, returns a full HTML body and offers a clear answer, I would look next at the answer itself and at competing evidence. If the fetch ends in a challenge or an empty body, I would not commission another rewrite first. The 60-minute plumbing check from groas follows that order, as does its always-on technical audit: permission, retrieval, rendering and extraction, checked against edge logs. Fix the failed gate, not the most marketable-sounding problem.

Checklist: inspect the page in fetch order

Use this on one page and one fetcher before generalizing across the site. Each item depends on the one above it: a polished passage is no help if the response never contains it. Record what you observe rather than marking a setting “done” and assuming the request succeeded.

1. Robots.txt for the relevant user agents. Check rules for ChatGPT-User, OAI-SearchBot, Claude-User and ClaudeBot on pages you want accessible; do not substitute a GPTBot check for all four.
2. Edge and WAF decisions for those requests. Identify the control issuing a challenge or block. A Super Bot Fight Mode Skip rule is not a fix for Bot Fight Mode; check the specific configuration in use.
3. Retrieval response. Follow the requested URL to the final page and inspect redirects, 403s, 429s, 500s and the returned body.
4. Readable HTML. View source without JavaScript and confirm the answer, relevant details and headings are present. A good title does not rescue an empty body.
5. Extractable answer. Put a direct, self-contained response under a clear heading; use readable text, lists or tables for details the answer depends on.
6. Verified versus unverified traffic. Check source IPs against vendor information before counting or acting on claimed crawler traffic. This is the item most often skipped, and it costs twice: you can misread a block as a visibility problem, then pay to repair content the verified fetcher never tried to read.

Apply the model to a different page

Take a pricing page that opens normally for a person but gets no useful live-answer exposure. I would not begin by writing a longer pricing explainer. First, I would identify which fetcher matters for the question, check its robots rule and look for a verified request. Then I would follow that request through the edge and inspect the returned status. Only after a successful response would I view the HTML without scripts and ask whether the price and its explanation are actually there.

If they are, the problem has moved beyond access: now the passage has to answer the buyer’s question clearly enough to be useful. If they are not, prose is not yet the repair. That is what the 15,515 requests taught me. A busy robots.txt is evidence of requests to a file, not evidence that assistants can read a page. Prove the fetch path first. Then make the page worth quoting.