LinkedIn
A perfect Lighthouse score will not help a page an AI agent cannot fetch, read or quote. Yet most technical audits start with the score and leave the access check for later. I used to do versions of that work by hand. I had the order wrong.
Rank every fix by one question: can an AI agent fetch, parse and quote this page during a live answer? On our own site, Claude-User requested robots.txt 1,247 times in 30 days to serve live answers, more than any content page, inside 4,125 verified AI fetches we kept after filtering spoofs (what 30 days of edge logs actually show). That is why this guide starts at the door. Access rules come before rendering, rendering comes before quotable structure, and schema comes last.
Why can a page score 100 and still miss AI answers?
Lighthouse measures a different job. Googlebot can render JavaScript like a browser, so a React headline may show up in Search Console. Live-answer fetchers can work from the first HTML response instead. If the headline arrives only after rendering, it may look fine in Chrome and be absent from what the fetcher reads (what an AI bot sees when it crawls your site).
The gap between rendered text and first-response HTML is the gap to check first. Compare Search Console URL Inspection with view-source on your top five money pages. If the answer appears only in the rendered version, a better performance score will not put it in the raw response. Fix in fetch order and you stop paying for work that never had a chance.
Step 1: Can the bot get in?
Most AI invisibility I have seen was not a content problem. It was a door problem. A bot that gets a 403 or a disallow on your money page never reaches your clever headings, and no amount of schema fixes that. Say you are spending $20k/month driving traffic to landing pages that live-answer bots cannot fetch. That content cannot win a citation from a fetch that fails.
Start with robots.txt, and read it by job, not just by vendor. Blocking a training crawler is not the same as blocking search or a user-requested fetch. OpenAI splits those jobs: GPTBot handles training, OAI-SearchBot builds the ChatGPT Search index, and ChatGPT-User fetches on demand when a user pastes a URL. Blocking GPTBot does not, by itself, remove you from ChatGPT Search (how OpenAI splits crawler jobs). Anthropic splits work across ClaudeBot, Claude-SearchBot and Claude-User. Google-Extended is another token worth checking: it governs Gemini training and grounding, not Search inclusion or AI Overviews eligibility (what Google-Extended actually controls). Allow the search and user-fetch agents on pages you want quoted. Block only what you mean to block.
Then check the edge. Your robots.txt can say Allow while the CDN says no. Cloudflare Bot Fight Mode, Super Bot Fight Mode, the AI-bot blocking toggle and custom WAF rules can return a 403, a 429 or a JavaScript challenge even when robots.txt allows the request (how edge rules overrule robots.txt). I have watched teams debug content for weeks while the edge log showed the same 403 on every live fetch. The CDN decides before your server ever sees the request.
A curl loop that swaps user-agent strings is a useful first check for status codes, but a user-agent string does not prove which bot made a logged request. Anyone can claim to be GPTBot. For traffic you count as real, reverse-DNS the IP, forward-confirm it, compare it with vendor-published ranges such as OpenAI’s per-bot JSONs, Anthropic’s bots.json and Perplexity’s JSONs, and bucket mismatched ASNs as spoofers (how to verify AI bot traffic). Fix a confirmed block by allowlisting verified bots, not by trusting a string a spoofer can copy.
Look in the right log, too. A CDN can serve cached pages without an origin hit; a WAF can block at the edge before the origin sees anything. Use edge logs to confirm whether verified agents actually received a 200, preferably across 30 days and at least two weeks (where to read AI crawler hits).
If you want the short version before touching code, run the check in Can AI Fetchers Read Your Page? A 45-Minute Check Before You Buy a Fix. It separates a door problem from a content problem, which is exactly what most 60-page audits miss.
Step 2: Can the bot read what it fetched?
A 200 status means the request got through. It does not tell you what the response body contained. If your price, spec table or answer exists only in the rendered DOM, a live-answer fetcher that reads first-response HTML leaves with an empty shell.
Test the response, not the browser view. Open view-source or run curl -A GPTBot and search the returned HTML for your headline (the 10-minute raw-HTML test). The mechanism is plain: text injected later by React, Vue or a tag manager is not in that first response. If the headline is missing from curl, fix how the page delivers it before rewriting the headline.
The same failure hides in copy loaded on scroll, tabs or accordions that need a click, and paywalls, cookie walls or interstitials that need dismissal. Server-side render the passages you want cited and leave the surrounding chrome to JavaScript. For AI Overviews, the page also needs to be indexed and snippet-eligible; noindex or nosnippet keeps it out (AI Overviews eligibility rules). The order stays the same: unblock the fetch, then ship the answer as plain HTML.
Step 3: Can the bot quote the answer?
Readable is not quotable. Give each answer a passage that makes sense on its own: a descriptive, question-shaped heading; the answer in the first one to three sentences; one idea per paragraph; and the subject named explicitly. AI systems retrieve passages, not whole pages, so a chunk that needs the previous four paragraphs to make sense is harder to lift (how AI passage retrieval works).
Use a list when you are giving steps and a table when you are comparing options. The extractor can lift those pieces whole. I used to tell clients a long pillar page would cover them. I was wrong. One mixed page can give the retriever five weak chunks where five tight passages would give it one strong quote. Rewrite the first 60 words under each H2 as if that heading and paragraph will travel without the page.
Then stop making the index choose between copies. Three near-identical location pages, a canonical pointing at the wrong URL, or a print version indexed beside the real one can split what little signal the answer earned. Set one canonical per answer and consolidate duplicate copies. Keep specific numbers in visible text, with units and dates: 31% lower CPA in 30 days gives a quotation more to carry than approximate savings. One answer, in one clear place, beats three competing versions of it.
Step 4: What should wait until after those fixes?
I like schema. I add it to most builds. What I stopped doing is selling it as the first AI visibility fix. Google says no special schema is needed for AI Overviews or AI Mode. In a controlled Ahrefs test of 1,885 pages that added JSON-LD against 4,000 controls, AI Overviews moved down 4.6%, AI Mode up 2.4% and ChatGPT up 2.2%: no compelling case for putting that work ahead of access (does schema help AI citations). Keep Organization, Product and FAQ schema for rich results and entity clarity. Do not put schema above a failed fetch.
The same order applies to newer machine-readable files. An Ahrefs analysis of 137,000 sites found that 97% of llms.txt files received zero traffic in May 2026, while OpenAI, Anthropic and Perplexity documentation focuses on crawler access through robots.txt (does llms.txt improve rankings). Sitemaps still matter for discovery, and a clean Markdown mirror does not hurt. Neither rescues a page the bot cannot fetch or read. Ship Steps 1 through 3 first. Add the extras afterward if you want completeness.
How do you verify a fix without trusting a dashboard?
Do not take a green check in a crawler tester as the final answer. Pull your CDN edge feed and Nginx or Apache combined logs for at least two weeks, ideally 30 days. Join fetched URLs to your sitemap to spot coverage gaps (where to read AI crawler hits). Filter for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, then verify IPs before counting hits. A spike from a hosting ASN that fails verification is noise, not demand.
Run these checks in order. Each one gates the next:
- Fetch check: For each money URL, list verified AI agents, edge status and bytes returned. Look for 200s from verified IPs. A run of 403s or 429s on ChatGPT-User means the WAF still owns the problem. Use a user-agent-swapping
curlloop as a diagnostic, not as proof of verified bot access. - Read check: Run
curl -A ChatGPT-Useragainst the same URL and search the response for the price, headline and first table row. If the returned HTML misses them, fix delivery before polishing the copy. - Quote check: Paste the URL into ChatGPT or Perplexity and ask for a summary with quotes. Inspect which passage it lifts. If it paraphrases without quoting, tighten the answer structure before touching schema.
One verified 200 plus visible answer text in curl tells you more than a hundred unverified hits in a traffic chart. It does not guarantee a citation. It tells you the page has cleared the first two gates.
How soon should you expect to see a change?
Access comes first because it gates every later fetch. Clear a bad disallow or WAF block, then look for a 200 on the next live-answer request in the edge log, potentially within a day or two. Do not use a rank tracker to confirm a door opened. Rendering fixes show up when the next fetch receives the revised raw HTML. Passage rewrites take longer, usually two to four weeks of prompt checks, because retrieval has to re-chunk the page and buyers have to ask the question again.
Use a different measure for each tier: edge status for access, curl output for rendering, and quoted answers for passages. Different fix, different clock.
When should you skip technical fixes and spend on content?
If your edge log already shows verified 200s from OAI-SearchBot, ChatGPT-User and PerplexityBot, and curl returns your price and headline as plain HTML, stop fixing the door. It is open. More access tickets will not earn the quote.
What you need then is material worth lifting: tighter answers, real tables, specific numbers and one clear page per question. I have seen small sites burn two months on schema plugins when they had four thin pages and no original data. No crawler setting creates demand for a quote. If access and rendering pass, move the budget to content.
Do one thing today. Open robots.txt, run curl -A ChatGPT-User against your top revenue URL, and search the returned HTML for its headline. If access or the answer fails that check, fix it before approving anything else. If you want the access, rendering and passage work handled without chasing developers, that is what groas runs inside its SEO and AI Search sprint: the technical fixes holding you back, then pages built to be quoted. Start with the fetch. Everything else waits behind it.