74% of Our robots.txt AI-Bot Hits Weren’t Verified. Why Are Agencies Reporting Raw Counts?

Only 26% of claimed AI-bot hits on our robots.txt file were verified. Over 30 days on harborfinch.com, that file logged 15,515 raw hits from User-Agents claiming to be AI bots. Cloudflare verified 4,047. If an agency puts the first number in a client’s AI-visibility report without showing the second, the report is selling noise as attention.

The same split appeared elsewhere, though not at the same rate. The homepage had a 30% verified share. One blog post had a 97% verified share and 435 recorded live ChatGPT-User and Claude-User fetches. Treat those paths as one domain-wide bot-traffic number and you erase the useful part of the story.

A User-Agent is a claim, not an ID card. Any script can put GPTBot in a header. Cloudflare’s verification gives us a way to separate independently confirmed bots from requests identified only by their headers; filtering for http.user_agent contains GPTBot alone does not do that. The reporting question is not just how many bots knocked? It is which requests verified, and which pages did live-answer fetchers request?

What the 30-day log split can—and cannot—show

I am looking at harborfinch.com edge logs, not GA4. Crawlers that fetch HTML without running analytics JavaScript will not appear as visits in that setup. To inspect their requests, you need server or CDN logs broken out by path.

For each path where the rollup allows it, I use three counts:

1. Raw hits: requests whose User-Agent claimed an AI-bot identity.
2. Verified hits: requests Cloudflare independently identified as coming from the claimed bot operator.
3. Live-answer fetches: requests from user-triggered fetchers, reported separately from bots that collect or index content.

Those counts answer different questions. A verified training crawler has really fetched a page, but that does not mean anyone saw the page in an answer. A live-answer fetch tells us the URL was requested in a user-triggered context; it does not prove the page was quoted, cited or responsible for pipeline. Nor does a raw count become more meaningful because someone gave it a polished chart.

GPTBot and ClaudeBot collect pages for later use. ChatGPT-User, Claude-User and Perplexity-User are the live-fetcher distinction that matters here. An agency report should not put them in the same bucket and call the sum AI visibility. That label skips the mechanism a client is paying to understand.

Finding 1: the icon folder exposes the raw-count problem

Multiple SVG files each recorded hundreds of raw AI-bot hits and only single-digit verified hits. In that folder, roughly 98% of the claimed hits went unverified. These were icon files, not the HTML pages someone would inspect to understand our content. They are a useful warning sign precisely because they are easy to overlook in a domain total.

I cannot tell from an unverified row who sent the request or why. I can tell that a report counting those rows as confirmed AI-crawler attention is claiming more than the data supports. A script can borrow a familiar User-Agent; an icon request does not become an answer-engine endorsement because the header says GPTBot.

A separate controlled test illustrates the same identification problem: on a new, unpromoted domain, 6 of 33 claimed live-assistant fetches and 107 of 799 Googlebot claims verified. It also found claimed ChatGPT-User requests looking for .env and config.json rather than content. That test is not harborfinch.com data, and I would not apply its rates to our logs. It does show why a User-Agent-only tally deserves scrutiny.

Exclude static assets from the client-facing content total. Keep them available for diagnosing traffic, but do not let repeated icon requests inflate a visibility claim.

Finding 2: the busiest paths shrink under verification

Source: 30 days of harborfinch.com edge logs. Verified share is verified hits divided by raw claimed hits for each path. The live-fetch column is a separate classification, not proof that every listed fetch also passed the verification check.

Both URLs invite broad traffic. Crawlers request /robots.txt to check rules. The homepage is the default place for a generic visitor or bot to start. That makes each path busy before we know who made the requests. On /robots.txt, 15,515 claimed hits become 4,047 verified ones. On the homepage, 6,330 become 1,916. Raw volume is not a substitute for verified identity.

The live-fetch column needs an even narrower reading. The logs record 1,231 Claude-User fetches of the rules file and 740 ChatGPT-User fetches of the homepage. A rules-file request does not mean Claude used /robots.txt as answer content. A homepage fetch does not establish that ChatGPT cited the homepage. Those rows tell us what the fetchers requested, not what appeared on a user’s screen.

I used to treat rising raw crawl volume as a sign of growing AI interest. I would not put that interpretation in a client report now. It tells us the server received more requests. Verification and page type have to come next.

Finding 3: the post and the index have opposite jobs

Source: the same 30-day harborfinch.com edge-log rollup. The blog-index verification split is unavailable here; its raw count should not be read as a verified count.

The post is a dated explainer with named changes, tight sections and passages a fetcher can retrieve directly. The blog index is a list of links and excerpts. It can help a crawler find a URL, but it does not do the same job as the article at that URL. That distinction makes the path-level comparison more useful than the domain total: the post recorded 435 live fetches; the index recorded 4.

The post’s 697 verified hits out of 718 raw claims also show that low verification is not a universal condition across this site. Different URLs produced different mixes of requests. The rollup does not provide a verification cross-tab for the 435 live fetches, so I will not call all 435 verified live fetches. It records a strong verified share for the page and, separately, substantial live-fetch activity. That is enough to decide which URL deserves a closer look. It is not enough to declare 435 citations.

Report both paths as bot traffic and the index looks busier: 1,160 raw hits against the post’s 718. Report the available verification split and live fetches per page, and the post becomes the clear place to investigate. The next content decision depends on that difference.

Why a domain-wide AI-visibility score misses the decision

The reports I see from agencies tend to lead with whichever total looks largest: bot hits, mentions or share of voice across a set of engines. I understand the temptation. One number fits neatly at the top of a slide. It also lets unlike observations hide inside an average.

Seer’s January 2026 discussion makes a related point about visibility scores: answers change with the prompt and the person, and engines do not publish rankings or prompt volume for this purpose. In its client interviews, 44% of real prompts already included brand names. A tracked mention may be worth examining, but the score alone does not tell a client what the reporting agency changed or what happened on the client’s site.

A separate dataset covering 924 small-business sites from September 6 to October 5, 2026 reported a median of 3.1 AI requests per Googlebot request and about 1,050 AI hits for the month. Its traffic breakdown was 11.3% live user-triggered fetches, 69.9% training and 18.8% search index traffic. Those are that dataset’s figures, not a rate I can apply to harborfinch.com. Their value here is the separation: a training request and a live request should not be sold as the same event.

On our site, the distinction becomes concrete at the page level. We recorded 740 live fetches on the homepage, 435 on one post and 4 on the blog index. We also recorded 1,231 Claude-User requests for /robots.txt, which belong in a technical-traffic view rather than a list of content successes. Roll those together and the client gets a larger number with less meaning.

I would split a report by page cluster before writing a recommendation: content pages, the homepage, discovery pages such as the blog index, and technical paths such as the rules file. Then I would show raw and verified counts where both exist, followed by live fetches and the named fetcher. That layout makes it harder to mistake crawler housekeeping for interest in a page someone might actually read.

It also matters for bot controls. OpenAI operates separate bots, and blocking GPTBot for training is not the same as blocking OAI-SearchBot for retrieval. If a report cannot distinguish which bot fetched which path, it is a poor basis for deciding what to allow or block. Keep bot identity and page purpose visible before you change a rule.

Here is the test I would run before putting an agency logo on an AEO report:

1. Remove static assets from the content view.
2. Put raw and verified hits beside each other for every path with an available split; mark missing splits as unknown.
3. Sort content URLs by live-answer fetches, with the fetcher named.
4. Keep technical requests, especially /robots.txt, separate from pages that could supply answer content.

Our post’s 97% verified share, the homepage’s 30% and the icon folder’s roughly 2% tell three different stories. One domain average would bury all three.

What I would demand before white-labeling an AEO report

If an agency puts its logo on a number, it owns the interpretation. Before I would white-label a platform for client retainers, I would ask it to show the request-level evidence behind a page-level summary: the claimed bot, whether it verified, the URL fetched and whether the request came from a live user-triggered fetcher. Raw, verified and unverified requests should not silently collapse into one column.

I would also ask how the platform handles multiple client domains and prompt limits. The pricing comparison in the draft’s source lists the following examples; tiers and included engines matter when comparing headline prices:

Those figures are a buying checklist, not proof that any one tracker has the best data. If you manage 20 client domains, per-domain charges and prompt caps affect what you can profitably report. Ask what the quoted tier includes before building a client deliverable around it.

Prompt trackers can also disagree about the same brand because their prompt sets, sample sizes, engines and collection methods differ. Compare share of voice within one tool over time, not as though two tools measured an identical universe. An edge-log view belongs next to the tracker: one samples where a brand appeared; the other shows requests to pages the site served. Neither, by itself, proves a sale.

groas for agencies pairs weekly Google and AI visibility tracking with content, technical fixes and citations, reported under the agency’s brand. I would still use one real client domain to inspect what a report exposes before putting that logo on it. If a platform shows only a score, I would not present the score as verified fetch evidence. The client deserves to know the difference.

The decision these logs support

This is one 30-day rollup from harborfinch.com, with a path list limited to the URLs that show the pattern. It is not a sitewide answer rate. Some paths lack a verification split in this rollup, and the live-fetch counts shown here are not cross-tabulated against verified status. Most importantly, a fetch is not a citation. I will not turn 435 live requests for one post into 435 appearances in answers.

The narrower conclusion is still decisive. On /robots.txt, 74% of claimed AI-bot hits did not verify. The homepage also lost most of its raw count under verification, while the post retained nearly all of its claimed hits and attracted far more live fetches than the blog index. Raw totals concealed those differences; page-level splits exposed them.

So I would stop sending clients a domain-wide raw bot-hit number as evidence of AI visibility. Send verified hits and recorded live-answer fetches per page, identify the fetcher, and separate content from technical paths. Mark what the logs cannot prove. For agencies choosing AEO reporting software, that is the filter: if it cannot make those distinctions, its biggest number is a vanity metric with a new name.