---
title: "15,436 AI Bot Hits, 4,125 Verified: What 30 Days of Edge Logs Actually Show"
description: "Thirty days of edge logs show how raw AI bot counts overstate verified crawler activity—and why live retrievals tell a different story from category-page crawls."
url: "https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/426a8b53-1546-496f-90d3-f876bc2c7e97.png"
published: "2026-10-20T05:26:57.541Z"
modified: "2026-10-20T05:26:57.611Z"
---

October 20, 2026 · 9 min read

# 15,436 AI Bot Hits, 4,125 Verified: What 30 Days of Edge Logs Actually Show

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![A tiny model checkpoint where an inspector lets few 'BOT'-tagged figurines through while most crowd a holding pen, showing most claimed AI crawlers are fakes.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/426a8b53-1546-496f-90d3-f876bc2c7e97.png)

In this article

1. [First, check who actually sent the request](#first-check-who-actually-sent-the-request)
2. [The path-by-path view changes the story](#the-path-by-path-view-changes-the-story)
3. [What the comparison supports—and what it doesn’t](#what-the-comparison-supportsand-what-it-doesnt)
4. [Why the measurement belongs before the slide deck](#why-the-measurement-belongs-before-the-slide-deck)
5. [Checklist: build a report a technical client can inspect](#checklist-build-a-report-a-technical-client-can-inspect)
6. [The decision the numbers support](#the-decision-the-numbers-support)

A chart showing 15,436 AI bot visits to our `robots.txt` file would look impressive in a client deck. Our edge logs verified just 4,125 of them. **Nearly 73% of the requests claiming to come from AI crawlers did not come from verified provider infrastructure.**

That is the problem with reporting User-Agent counts as AI visibility. On this path, the raw count was roughly 3.7 times the verified count. On asset files, the gap was much worse: 777 claimed AI bot hits, 13 verified. Meanwhile, the pages drawing the most requests we classified as live answer retrievals were dated changelog posts, not category hubs.

I looked at 30 continuous days of requests across our own edge infrastructure to separate three things agencies too often collapse into one graph: **claimed bot traffic, verified provider traffic, and live retrieval requests**. The distinction matters. A request to `robots.txt` is not interest in your brand; a verified crawl is not necessarily a response to a user’s question; and a live retrieval does not prove your page appeared in the answer. If an agency reports only the first number, it is reporting noise as progress.

## First, check who actually sent the request

A User-Agent is an unauthenticated string. A scraping script or vulnerability scanner can send a request while calling itself `GPTBot`; the header alone tells you nothing about who operates it. To verify a crawler, you need to check the connecting IP against the provider’s published network ranges or use forward-confirmed reverse DNS. [Arcjet’s explanation of bot spoofing](https://arcjet.com/learn/detect-bot-spoofing) walks through the distinction.

Other published observations point in the same direction, though they are not a substitute for checking your own traffic. [Centinel Analytica](https://www.centinelanalytica.com/bots/gptbot) reports that roughly 94% of inbound requests claiming to be GPTBot fail basic IP verification. In an audit of production logs across client domains, [Hey-Eye](https://hey-eye.gr/blog/verify-real-ai-crawlers-vs-spoofed-bots/) found that more than 90% of requests declaring frontier AI signatures failed its IP and reverse-DNS checks. Different samples, different methods. The useful lesson is narrower: **do not authenticate a crawler by reading the name it gave itself.**

![Server log display contrasting raw bot hits with verified crawler records.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e91bb391-85fb-4c89-9225-6e6480680d20.png)

### The `robots.txt` gap: 11,311 unverified hits

In our edge logs, `/robots.txt` received 15,436 requests carrying AI crawler User-Agents over 30 days. Matching the connecting IPs against provider network manifests left 4,125 verified requests and 11,311 unverified ones. Reporting the raw number as verified crawler attention would overstate it by 274%.

The path matters, too. `robots.txt` is a permissions file, not a page that answers a customer’s question. Scrapers and scanners have reasons to request it that have nothing to do with AI visibility. [Cypho describes](https://cypho.io/blogs/wearing-a-crawlers-name-ua-spoofing) vulnerability scanners forging crawler headers to get past defenses that trust those names. Whatever the unverified requests in our logs were doing, they do not belong in a client-facing count of authenticated AI crawler activity.

The practical rule is simple: **verify the sender before you interpret the visit.**

## The path-by-path view changes the story

Once I grouped the requests by URL type, the raw totals stopped looking like a single measure of interest. The table below comes from the same 30 days of our edge logs. “Verified” means the connecting IP passed our provider checks. “Live retrieval” is a request classification in those logs, not proof that a final answer cited the page.

| Path type                             | Raw claimed AI bot hits | Verified provider hits | Classified live retrievals | Verification rate |
| :------------------------------------ | ----------------------: | ---------------------: | -------------------------: | ----------------: |
| `robots.txt`                          |                  15,436 |                  4,125 |                          0 |             26.7% |
| Static assets (`.css`, `.js`, images) |                     777 |                     13 |                          0 |              1.7% |
| Homepage (`/`)                        |                   6,223 |                  1,893 |                        789 |             30.4% |
| Category hubs                         |                   1,420 |                    812 |                          0 |             57.2% |
| Changelog and technical guides        |                     842 |                    817 |                        751 |             97.0% |

![Chart comparing raw AI bot hits, verified provider hits, and classified live retrievals across five path types.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d886ab26-2e03-4be8-8b71-57126c26fa5e.png)

### Assets: nearly 60 raw hits for each verified hit

Static files gave us the starkest verification gap. Of 777 requests to asset paths carrying AI bot headers, **13 came from verified provider IPs**. The raw total was nearly 60 times the verified total.

A scraper that loads an HTML page may also request its linked stylesheets, scripts, and images while reusing the same User-Agent. Those requests can make a CDN chart look busy. They do not show that an answer engine found your logo file useful to a person asking a question. If an agency adds asset requests to its AI attention total, the metric gets bigger while the evidence gets weaker.

**Keep assets out of the brand-attention number.**

### Homepage: verification still leaves two kinds of activity

The homepage received 6,223 claimed AI bot hits. We verified 1,893, a 30.4% verification rate. But verification is only the first filter: our logs classified 1,104 of those verified requests as broad background activity and 789 as live retrievals.

That separation has a technical basis. [OpenAI documents distinct crawler roles](https://developers.openai.com/api/docs/bots), including `GPTBot` for training, `OAI-SearchBot` for search indexing, and `ChatGPT-User` for user-initiated retrieval. They should not be rolled into a single “people found us in AI” number. Against the 789 requests our logs classified as live retrievals, the raw homepage count was nearly eight times as large.

I would still stop short of calling those 789 answers, citations, or buyers. **A retrieval request shows that a page was fetched in a live-use context; the edge log does not show the final response.** That is a more useful metric precisely because its limit is clear.

### Changelogs: the live retrievals clustered on dated posts

The technical changelogs and dated guides reversed the pattern seen on assets and `robots.txt`. Of 842 claimed bot hits across those URLs, **817 were verified and 751 were classified as live retrievals**. Two changelog updates, covering Google Ads bidding logic and tracking changes, accounted for 467 and 284 live retrievals respectively.

Those figures show where retrieval activity concentrated during this month. They do not tell us the prompts behind every request or whether either post was quoted in a response. Still, the contrast is worth an editor’s attention: these specific, dated pages drew most of the live retrievals in this group. That is a stronger reason to inspect what they explain than a sitewide bot total ever would be.

**Keep the URL and its content type beside the count.** Otherwise the most useful pattern disappears.

### Category hubs: crawled, but not live-retrieved here

Our category hubs recorded 1,420 claimed AI bot requests, including 812 verified provider hits. They recorded **zero requests classified as live retrievals** during the same 30 days.

That does not prove an engine read those pages, judged the copy inadequate, and discarded them. A server log cannot reveal that decision. It does tell us that verified crawling and live retrieval were different outcomes on these paths in this period. Calling the 812 hits “AI answer visibility” would erase the distinction the report is supposed to make.

**Flag this gap for investigation, not an automatic rewrite order.**

## What the comparison supports—and what it doesn’t

The most striking split is not simply between high-traffic and low-traffic pages. It is between paths where claimed bot traffic largely failed verification and paths where verified requests were frequently classified as live retrievals. In this sample, `robots.txt` had 15,436 raw hits and no live retrievals. The changelog and guide group had 842 raw hits and 751 classified live retrievals. A single top-line “AI bot visits” graph would hide both facts.

Content specificity may help explain why the dated technical posts drew that activity while category hubs did not. A post about a particular bidding or tracking change has a more precise subject than a broad service overview. But the logs establish **where requests landed, not why an engine selected a page**. I would use the pattern to choose which URLs to examine and which reporting claims to challenge, not to announce a universal rule that answer engines never retrieve category pages.

The sample has limits. It covers one production domain, one 30-day window, and a site in the paid and organic search technology space. It cannot settle how an ecommerce category page performs for a shopping query. Nor does a month without a classified live retrieval prove that background crawling has no later value. What it does establish for this site is enough to change a report: **raw claimed hits, verified crawls, and live retrievals are not interchangeable numbers.**

## Why the measurement belongs before the slide deck

Standard client-side analytics will not give an agency the whole crawler picture. Tools such as Google Analytics 4 and Google Tag Manager depend on JavaScript running in a browser; a crawler requesting page content without running that script will not register as an ordinary pageview. [Similarweb’s log-file discussion](https://aisearch.similarweb.com/blog/log-file-analysis/) explains why server-side request data fills that gap.

That does not mean every client needs an elaborate new reporting estate before an agency can say anything useful. It means the agency needs access to request-level evidence it can authenticate and classify. Edge middleware is one approach: it can inspect the connecting IP and request before the traffic reaches the origin. The draft [ai-crawler-intercept pipeline](https://github.com/Unsourced-Founder/ai-crawler-intercept) illustrates that approach. The implementation choice matters less than the audit trail behind the number handed to a client.

![Edge proxy diagram showing a crawler IP check before the request reaches an origin server.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/b7c5c070-ff84-4ea2-887d-f3856b1c7fa0.png)

**If you cannot show how a bot count was verified, do not label it verified traffic.**

## Checklist: build a report a technical client can inspect

I would ask for one compact report before accepting a chart about AI visibility. It should separate sender verification, request purpose, and the URLs involved. Keep each figure close enough to the underlying request data that someone can check it; a polished graph is not an audit trail.

1. **Verify the connecting IP.** Check provider network ranges or forward-confirmed reverse DNS; a User-Agent string is a claim, not an identity.
2. **Remove `robots.txt` and static assets from content-attention totals.** Report them separately if they matter operationally, but do not count permission-file or stylesheet requests as page interest.
3. **Separate background activity from live retrievals.** Keep crawler roles and request classifications visible rather than combining every verified hit into one answer-visibility figure.
4. **Show the requested URLs by content type.** A homepage hit, category-hub crawl, and retrieval of a dated technical post tell different stories.
5. **Flag verified-but-not-live-retrieved pages for review.** This is the step most easily skipped: without it, a client can keep paying for work against pages that look active in crawler totals but show no live retrievals in the reporting window.

## The decision the numbers support

An agency selling Answer Engine Optimization should not bill for a monthly presentation that treats unauthenticated headers as evidence of demand. On our `robots.txt` path alone, that would have meant presenting 15,436 claimed hits where only 4,125 passed verification. On asset paths, it would have meant treating 777 requests as attention when just 13 were verified. Those are not small reporting errors. They change the story the client hears.

This is also where the operating model matters. An account manager manually moving raw logs into a slide deck can produce a neat report while missing the distinction between a scanner, a background crawl, and a live retrieval. We built [groas](https://groas.com/) around the alternative: continuous search execution by specialized models, with a human strategist setting direction and remaining accountable for outcomes. For partners, [groas for agencies](https://groas.com/for-agencies) offers that execution under the agency’s brand rather than leaving every client report and follow-up action in a manual queue.

The decision I would make from these 30 days is not “publish more changelogs” or “delete the category hubs.” The data does not support either shortcut. **Stop reporting raw bot headers as AI visibility.** Verify the request, distinguish live retrieval from background activity, then inspect the pages each type of traffic reaches. If a client is paying for search impact, that is the minimum standard the numbers can defend.

## Frequently Asked Questions

### How can I determine if an AI crawler request is legitimate instead of spoofed?

A User-Agent string is not authentication and does not prove a crawler's identity. To verify a crawler, you must check the connecting IP against the provider's published network ranges or use forward-confirmed reverse DNS.

### Why is it incorrect to use a raw AI bot count as a measure of brand visibility?

Raw counts mix claimed bot traffic, verified provider traffic, and live retrieval requests, which are distinct categories. Reporting only the raw count essentially reports 'noise' rather than genuine progress.

### Should static files and robots.txt requests be counted when measuring AI interest in a brand?

No, static assets and robots.txt should be excluded from the brand-attention number. These are permission files or linked media, not content that answers a user’s question.

### What is the difference between a verified retrieval and a final answer citation?

A live retrieval request only shows that a page was fetched in a live-use context. The edge log does not provide evidence of the final response, meaning the retrieval does not prove the page was cited by an answer engine.

### What kind of content draws the most live retrieval activity from AI crawlers?

Live retrieval activity tends to cluster on specific, dated pages such as changelog posts and technical guides. These specific topics usually have a more precise subject than a general service overview.

### What should be included in a report to accurately measure AI visibility?

An accurate report must separate sender verification, request purpose, and the requested URLs. This audit trail allows someone to check the figures against the underlying request data.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#webpage","url":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t","name":"15,436 AI Bot Hits, 4,125 Verified: What 30 Days of Edge Logs Actually Show","description":"Thirty days of edge logs show how raw AI bot counts overstate verified crawler activity—and why live retrievals tell a different story from category-page crawls.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/426a8b53-1546-496f-90d3-f876bc2c7e97.png"},"breadcrumb":{"@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#breadcrumb"},"dateModified":"2026-10-20T05:26:57.611Z","mainEntity":{"@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"15,436 AI Bot Hits, 4,125 Verified: What 30 Days of Edge Logs Actually Show","item":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#article","url":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t","mainEntityOfPage":{"@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"15,436 AI Bot Hits, 4,125 Verified: What 30 Days of Edge Logs Actually Show","description":"Thirty days of edge logs show how raw AI bot counts overstate verified crawler activity—and why live retrievals tell a different story from category-page crawls.","datePublished":"2026-10-20T05:26:57.541Z","dateModified":"2026-10-20T05:26:57.611Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/426a8b53-1546-496f-90d3-f876bc2c7e97.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"verified, bot, live, live retrievals, requests, hits, logs, retrievals","wordCount":1925,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/73-of-the-ai-bot-traffic-on-our-robots-t#faq","mainEntity":[{"@type":"Question","name":"How can I determine if an AI crawler request is legitimate instead of spoofed?","acceptedAnswer":{"@type":"Answer","text":"A User-Agent string is not authentication and does not prove a crawler's identity. To verify a crawler, you must check the connecting IP against the provider's published network ranges or use forward-confirmed reverse DNS."}},{"@type":"Question","name":"Why is it incorrect to use a raw AI bot count as a measure of brand visibility?","acceptedAnswer":{"@type":"Answer","text":"Raw counts mix claimed bot traffic, verified provider traffic, and live retrieval requests, which are distinct categories. Reporting only the raw count essentially reports 'noise' rather than genuine progress."}},{"@type":"Question","name":"Should static files and robots.txt requests be counted when measuring AI interest in a brand?","acceptedAnswer":{"@type":"Answer","text":"No, static assets and robots.txt should be excluded from the brand-attention number. These are permission files or linked media, not content that answers a user’s question."}},{"@type":"Question","name":"What is the difference between a verified retrieval and a final answer citation?","acceptedAnswer":{"@type":"Answer","text":"A live retrieval request only shows that a page was fetched in a live-use context. The edge log does not provide evidence of the final response, meaning the retrieval does not prove the page was cited by an answer engine."}},{"@type":"Question","name":"What kind of content draws the most live retrieval activity from AI crawlers?","acceptedAnswer":{"@type":"Answer","text":"Live retrieval activity tends to cluster on specific, dated pages such as changelog posts and technical guides. These specific topics usually have a more precise subject than a general service overview."}},{"@type":"Question","name":"What should be included in a report to accurately measure AI visibility?","acceptedAnswer":{"@type":"Answer","text":"An accurate report must separate sender verification, request purpose, and the requested URLs. This audit trail allows someone to check the figures against the underlying request data."}}]}]}
```