On this page
- Why JavaScript analytics can't see AI crawlers
- What a crawler request is (and isn't): fetches, referrals, citations and recommendations
- How to track AI crawlers without JavaScript: server logs, CDN analytics and log drains
- A log-reading example: grep GPTBot, ClaudeBot and PerplexityBot in an access log
- User agents are claims: verifying a bot with published IP ranges
- What to do with the numbers, and how to block AI crawlers if you need to
- How Revnu counts requests on the pages it serves
- Start with one grep, then count what matters
You can track AI crawlers without JavaScript by reading the request record your server or CDN already keeps: filter access logs for the user agents GPTBot, ClaudeBot and PerplexityBot, then verify the source IPs against each provider's published ranges.
This matters because AI crawler tracking can't live in a browser. These bots fetch your HTML and leave. They never run your analytics tag, so Google Analytics, PostHog and every other script-based tool shows nothing. The log line is the only evidence.
That evidence is narrow, and this guide is honest about it. A fetch is not a referral, a citation or a recommendation. A user-agent string is a claim, not proof. Below: why scripts miss crawlers, what a request actually proves, three places to collect the data, a grep example, how to verify a bot, what to do with the numbers, how to block crawlers, and where Revnu fits.
Why JavaScript analytics can't see AI crawlers
JavaScript analytics can't see AI crawlers because a crawler requests the page, reads the HTML response and moves on without executing any script. Your tag never fires, so no event is recorded.
That's the whole mechanism. A browser downloads your page, runs the analytics snippet, and the snippet phones home. A crawler stops at step one. The only place its visit exists is the web server, the CDN or the load balancer that answered the request.
It also explains a common surprise. Your dashboard says no AI traffic, yet your logs show hundreds of GPTBot hits. Both are true. The dashboard measures people with browsers. The log measures requests.
If you can't or won't add a script tag, you lose nothing by skipping it. Server-side records are the better source for crawler questions anyway.
What a crawler request is (and isn't): fetches, referrals, citations and recommendations
A crawler request is a fetch: a bot asked your server for a URL and got a response. That is all the log proves.
Four things get confused, so separate them:
Fetch: a bot requested the page. This is what logs show.
Referral: a person clicked from an AI answer to your site. That appears as a normal browser visit with a referrer, and your script-based analytics can see it.
Citation: an AI answer linked or named your page as a source. A fetch doesn't guarantee one.
Recommendation: an AI answer told a buyer to choose you. Neither a fetch nor a citation guarantees this.
The vendors also run different bots for different jobs. OpenAI documents separate agents for training, search and user-triggered fetches (OpenAI). Anthropic does the same: ClaudeBot is its training crawler, and it uses other robots to search and to retrieve pages (Anthropic). Perplexity publishes its own bot list too (Perplexity).
So a GPTBot hit means something different from a ChatGPT-User hit. The first is a training-related crawl. The second is a fetch triggered by someone's request, which is closer to a human asking about your page, though still not a citation. Count them separately.
How to track AI crawlers without JavaScript: server logs, CDN analytics and log drains
There are three places to collect crawler requests without a script: your origin server's access log, your CDN's analytics, and a log drain that ships logs somewhere you can query.
1. Origin access logs. Nginx and Apache already write one line per request with the timestamp, path, status code, source IP and user agent. Make sure your log format includes the user agent (the common "combined" format does). This is the cheapest option and works on any stack you control.
2. CDN analytics. If a CDN sits in front of your site, many fetches never reach your origin because they are served from cache. Your origin log undercounts. Cloudflare's AI Crawl Control provides crawler monitoring for site owners, though it is not available with all features on all plans: free-plan analytics are limited to 24 hours and rely on user-agent identification, while Bot Management enables verified detection and more advanced analytics (Cloudflare). Its Crawlers and Metrics views show request counts per crawler, and the GraphQL Analytics API exposes the same data programmatically. One caution: Cloudflare documents its verified-bot filter as a Bot Management feature, while a plain user-agent filter works on every plan but can be spoofed.
3. Log drains. On hosts like Vercel or Fastly, stream request logs to a destination you control (object storage, a log platform, a database) and query them there. This is the route when you want history beyond what the dashboard keeps.
Pick by where your traffic is actually served. If a CDN fronts the site, start there. If not, start with the origin log. Either way, you're counting requests, which is exactly what the question asks.
A log-reading example: grep GPTBot, ClaudeBot and PerplexityBot in an access log
To identify GPTBot, ClaudeBot and PerplexityBot in nginx or Apache access logs, search the user-agent field with a case-insensitive pattern:
grep -Ei 'GPTBot|ClaudeBot|PerplexityBot' access.logAdd the search and user-fetch agents if you want the full picture: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and Perplexity-User. Check each vendor's documentation for the current list, since agent names change.
Here is an illustration of what a matching line looks like. The values are made up:
203.0.113.45 - - [14/Mar/2026:09:12:07 +0000] "GET /blog/example-post HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"Count requests per bot:
grep -Eoi 'GPTBot|ClaudeBot|PerplexityBot' access.log | sort | uniq -c | sort -rnCount which pages each bot fetched, and with what status:
grep -i 'GPTBot' access.log | awk '{print $7, $9}' | sort | uniq -c | sort -rn | headRead the output for three things. Which pages get fetched, since bots favour some sections over others. Which status codes come back, because a pile of 403s or 404s means the bot is hitting walls or dead URLs. And how the volume moves by day.
The output can't tell you whether any answer engine used those pages, or whether the requests really came from the vendor. That second point is next.
User agents are claims: verifying a bot with published IP ranges
A user agent is a claim the client makes about itself, and anyone can send any string. A scraper can call itself GPTBot in one line of code.
Verify by checking the source IP against the provider's published ranges. OpenAI, Anthropic and Perplexity each document their crawlers, and Perplexity points to official JSON endpoints for current IP ranges and recommends combining user-agent matching with IP verification (Perplexity). Always use the most current ranges from the vendor's own page rather than a copied list.
A workable routine:
Extract the unique IPs for each claimed bot from your logs.
Compare them with the vendor's current published ranges.
Report verified and unverified requests as separate counts.
Unverified doesn't mean fake. Ranges change, and your CDN may rewrite the client IP, so check which header carries the real source address. But don't fold unverified hits into a headline number without saying so.
The same logic applies to rules you write. If you allow or block by user agent alone, a spoofer walks around the rule. Pair user agent with IP in any firewall rule that matters.
What to do with the numbers, and how to block AI crawlers if you need to
Use crawler counts as a health signal for access, not as a performance score. They tell you whether bots can reach the pages you care about and how often they return.
Useful actions:
Fix what bots can't read. Repeated 403, 404 or 5xx responses to a crawler on an important page is a defect you can repair today.
Compare bot types. Search-agent and user-fetch hits on a page suggest it's being looked at to answer questions. Training-crawl hits alone say less.
Watch for sudden changes. A drop to zero after a deploy usually means a firewall rule or a robots.txt change blocked them.
Pair with the outcomes that matter. Check AI referral visits in your normal analytics and check whether answers cite you. Those are separate measurements.
To block AI web crawlers, use robots.txt rules naming each bot's user-agent token, for example a User-agent: GPTBot group with Disallow: /. The vendors treat their agents separately: blocking OpenAI's training crawler doesn't by itself block its search agent, and Anthropic draws the same line between its training crawler ClaudeBot and its search and user-fetch robots (OpenAI, Anthropic). Anthropic also notes that rules naming ClaudeBot directly take priority over its general signal. Decide per agent: you may want to block training while allowing search.
Robots.txt is a request, not enforcement. For hard blocking, add a WAF rule on user agent plus verified IP, in Cloudflare or your own firewall. If you block everything, you also remove yourself from any answer that needs to fetch your page.
How Revnu counts requests on the pages it serves
Revnu is the growth agent that finds customers across SEO, AI search, outbound and paid ads, and its AI crawler tracking is deliberately narrow: it records crawler requests, covering search and user-fetch bots, on pages Revnu serves, such as Revnu-hosted blog posts.
Three limits keep the numbers honest. They're request counts, not unique agents. Bot identities aren't independently verified. And pages hosted on your own site aren't covered, so a marketing site Revnu doesn't serve won't appear in those counts.
That makes the Revnu figure a clean answer to one question: are bots fetching the blog posts Revnu publishes for you? It sits next to the rest of Revnu's AI search work, which tracks how your business appears in answers from ChatGPT, Claude, Perplexity and Gemini. For crawler activity on the rest of your site, the log methods above are the route. If you want the wider picture of what to look for in AI search tooling, see 7 Best AI Marketing Agents for SaaS (2026), and for the answer-mention side of the problem, Get recommended by ChatGPT and AI search.
Start with one grep, then count what matters
Run the grep command on last week's access log today, split verified from unverified IPs, and you'll know more about AI crawler traffic than any script-based dashboard can tell you. Keep the claim small: a fetch proves access, nothing more. For Revnu-hosted blog posts, Revnu records those request counts for you. Sign up at revnu.com/pricing, where Pro is $99 a month, and the numbers sit beside your AI search tracking.
Written by
Co-founder and CEO, Revnu
