Your AI SEO Audit Checked Schema. Did It Check Whether Bots Could Load the Page?

Your AI SEO audit recommended schema markup. It never checked whether ChatGPT-User or Claude-User could load the page.

I keep seeing the same deliverable: 80 pages, a nice cover, sections on FAQ schema and meta descriptions and E-E-A-T, and not one line showing what a bot got back when it tried to fetch the site. That is not an AI visibility audit. That is a 2015 SEO audit with the word AI pasted on top.

Here is why it bothers me. In groas edge logs for the last 30 days, robots.txt was the most-fetched file by live-answer bots, with 1,227 Claude-User live-answer fetches hitting that file. The bots check the door first. Most audits never look at the door.

First, the audit sells you a nicer doorbell

The padding usually opens with “Add FAQ schema.” An Ahrefs matched-control study of 1,885 pages that added JSON-LD versus about 4,000 controls found ChatGPT citations up 2.2% and AI Mode up 2.4%, both noise, with AI Overviews down 4.6%. Google says there is no special schema needed for AI Overviews or AI Mode beyond being indexed and eligible for a snippet.

Then someone points out that about 53% of AI-cited pages carry schema, roughly three times the non-cited rate. That is the correlation trap. Maintained sites tend to do lots of things well. Finding schema on a cited page does not mean schema earned the citation, and it certainly does not tell you whether an assistant can fetch your page.

Next comes “improve E-E-A-T signals,” which often translates to: write like you have done the work before. Fine advice, if there is work worth describing. My favorite is “optimize meta descriptions for LLMs.” Live-retrieval tests have found assistants extracting visible HTML while ignoring JSON-LD and microdata; one vendor test reported six of seven systems, including ChatGPT, Claude and Perplexity, could not reliably fetch JSON-LD. None of that makes readable pages or useful descriptions worthless. It does make them a strange first order of business when nobody has tested access.

The audit told you to repaint walls the bot never walks past.

The fetch happens before the citation

Bots fetch. They do not read your audit. In our logs, robots.txt outranked every money page for live-answer bots. A fetch that gets a 403, a redirect loop or an empty shell of JavaScript cannot quote the useful text you thought you published. Your FAQ schema may be immaculate. The page still has to arrive.

JavaScript makes this easy to miss. A Vercel and MERJ network study found none of the major AI crawlers it examined rendered JavaScript, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Downloading a script is not the same as executing it. In that data, ChatGPT fetched JS on 11.50% of requests and Claude on 23.84%, yet executed none of it. If your main text appears only after a script runs, those fetchers do not receive that text in the HTML.

I used to tell clients a green check in a head-tag test meant they were safe. I was wrong. A test can find the title and meta tags while missing the empty body underneath. One practitioner check of 50 sites found 20 of 32 judged sites sending a crawler 1 to 11 words of body content against a 631-word median for humans. The browser shows a page. The fetcher gets an empty root div.

You can start with an unglamorous check: View Source and search for a sentence from the middle of the page. If it is missing, the content may depend on JavaScript. Curl the HTML, too. Neither check replaces a real bot fetch, but both tell you more about what arrived than another page of markup recommendations. No audit that skips the response deserves AI on the cover.

“AI bot” is not one bot

Here is another distinction the PDF tends to flatten. GPTBot is used for training, OAI-SearchBot for ChatGPT search indexing, and ChatGPT-User for a user-triggered fetch. Those are different jobs. A line that says “check robots.txt for AI” does not tell you which token was checked, what it was allowed to reach or which visibility problem the rule might cause.

Anthropic has its own set of tokens. ClaudeBot, Claude-SearchBot and Claude-User honor robots.txt, so a Disallow on Claude-User can stop live retrieval even if ClaudeBot is allowed. Check rules for each relevant subdomain as well. It is perfectly possible to examine the main site, declare it open, and miss the rule on the subdomain serving the page you want quoted.

I have seen the maddening version: ClaudeBot allowed, Claude-User blocked, FAQ schema recommended anyway. The fix starts with a less impressive question than “Are we AI-ready?” Which bot, which fetch, which subdomain? Make the audit answer that before it scores your markup.

The landing-page change nobody mentioned

This reminds me of paid search. I have watched a Quality Score tank after someone swapped a landing page without telling the person managing the account. The keywords and ads had not become mysterious overnight. The page had changed. Looking at what the crawler could reach was more useful than admiring the campaign report.

The same blind spot now sits between a public page and an AI fetcher. Cloudflare changed its default to block AI crawlers unless owners opt in. Your robots.txt can say allow while the edge says block. A firewall rule can outlive the scraping panic that prompted it. A JavaScript framework upgrade can change what lands in the initial HTML. Each change puts the same question back on the table: what did the bot get back today?

I wrote about the paid-search version in why grader-score audits do not fix campaigns. A score does not fix anything. A fetch log at least shows you what broke. If the page changes on Thursday, Monday’s tidy audit is not a control system.

Eighty pages are easier to bill for than a fetch test

Fetching your own robots.txt with a bot user-agent takes minutes. Curling a page with JavaScript off takes less. Neither produces a handsome PDF. Schema tables and E-E-A-T maturity matrices do, even when they never establish whether the assistant received a readable response.

That is the grievance. I am not asking auditors to hate schema or abandon editorial judgment. I am asking them to put the dependency first. If access fails, the arguments about wording and markup are happening on the wrong side of a locked door. What the deck calls “AI readiness,” I call two people doing the same 2015 job while the door stays locked.

The six checks to demand before another audit

Run these in order, from access to the ongoing watch. Keep the result for each page you care about; “looks fine in my browser” is not a result from a bot fetch.

1. Check robots.txt by user-agent. Inspect GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and Claude-User separately, including relevant subdomains.
2. Check the CDN or WAF response. A permissive robots.txt does not override a firewall that returns a block.
3. Check the HTML without JavaScript. Find a mid-page sentence in View Source; if it is absent, investigate what the fetcher receives.
4. Fetch your top 10 money pages with ChatGPT-User and Claude-User user-agents. Record status codes, redirects and the body returned, not just the browser view.
5. Check that the sitemap is current and fetched. Look for an actual bot request in the server logs rather than treating a submitted sitemap as proof.
6. Name who gets alerted when a key page changes. Without that owner, a deploy or new rule can undo the first five checks before anyone notices the lost citations.

That last item is the one everyone skips. A page can pass on Monday and fail after a Thursday deploy, a plugin update or a new WAF rule. If nobody watches status codes and the amount of readable text returned from money pages, the first sign may be a drop in traffic. Explanation comes later, usually in a meeting with too many slides.

The fix is a watch, not another PDF

An audit is a snapshot. The fix I want is an access check that keeps running, with someone accountable for acting when it fails. Fetch the important pages as the relevant bots. Record what came back. Notice when a 200 becomes a 403 or the readable body disappears. Then investigate the rule, template or deployment that changed it. That is dull work. Dull work is exactly what should not wait for the next quarterly presentation.

That principle is why I prefer continuous execution to another pile of recommendations. groas is built around autonomous search work, logged actions and a named human strategist who sets direction and owns the outcome. That does not make a blocked CDN rule magically disappear. Someone still has to identify the barrier and change the right thing within the right guardrails. But a report that merely labels the site “AI-ready” while nobody checks the next fetch is not even trying to solve this problem.

For landing pages, I would put server-rendered, visible HTML first: give the fetcher the actual answer near the top, in a structure it can read. Schema can come after the page is accessible and useful. If you want to run the checks yourself, Can AI Fetchers Read Your Page? A 45-Minute Check Before You Buy a Fix walks through robots.txt, bot user-agents and no-JavaScript checks. Start there before paying someone to describe your headings back to you.

Fix the door before you repaint the house

I do not hate schema. I run JSON-LD on pages I control. I hate the order of operations in these audits. They sell you curtains while the front door is nailed shut. Fetch first, render second, monitor access after that. Then argue about FAQ markup and E-E-A-T wording. A vendor who reverses that order is billing you for what is easy to write, not what decides whether your page can be read.

This will not matter equally to every site. If what you need people to find lives inside a logged-in app or a marketplace listing rather than a public page, fix the place they can actually reach. For public URLs you want an assistant to quote, though, the fetch is the funnel. No accessible page, no amount of polish inside the page gets a turn.

If you already paid for one of these audits, do not bin it. Use whatever is useful. Then send the vendor one email: show me the fetch results for my top 10 pages as ChatGPT-User and Claude-User, with status codes and readable word counts. Keep those results and check again after every deploy. If the vendor cannot show you what the bot received, you know what the 80 pages left out.

They audited the document. They did not audit the door.