How to Audit and Analyze AI Crawler Traffic on Your Website

Key takeaways

• Server logs provide the clearest view of which AI crawlers are accessing your site and how they behave
• Training crawlers and retrieval crawlers serve different purposes and should be managed separately
• Status codes, crawl frequency, and URL patterns can reveal technical barriers affecting AI access
• robots.txt and related controls can shape crawler access at the bot and content level
• Crawl health becomes more useful when compared with AI citation visibility and referral traffic

AI search visibility starts well before an answer is generated. AI systems need access to web content they can retrieve, index, or fetch when responding to a user.

Server logs show exactly how that access happens. They record which crawler requested a page, which URL it reached, when the request happened, and how your server responded. 

That makes them one of the clearest places to diagnose a basic question many teams still struggle to answer: Can the AI systems you care about actually access the content you want them to surface?

The volume is becoming difficult to ignore. In Cloudflare’s analysis of crawler traffic, GPTBot grew from 2.2% of traffic among a cohort of AI and search crawlers in May 2024 to 7.7% in May 2025, with raw requests increasing 305%.

In separate research with MERJ, Vercel found that GPTBot and Claude together made about 939 million requests across its network in one month, roughly 20% of Googlebot’s volume. Vercel also found that none of the major AI crawlers it tested rendered JavaScript.

Auditing that activity gives technical SEO and web teams a way to see those differences on their own site, determine what needs attention, and take the right action.

Which AI crawler user-agents should you track?

Start with the user-agent strings appearing in your access logs.

The important distinction is purpose. Some crawlers collect content that may contribute to future model development. Others support search indexing or user-triggered retrieval. Those functions can have very different implications for an AI visibility strategy.

AI crawler and retrieval user-agents to monitor in server logs:

A few details matter here.

OpenAI explicitly separates GPTBot from OAI-SearchBot. Its crawler documentation says publishers can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training. OpenAI also says ChatGPT-User is associated with specific user actions and is not an automatic web crawler.

Anthropic now documents three separate bots: ClaudeBot for potential training use, Claude-SearchBot for search, and Claude-User for user-directed retrieval.

Perplexity makes a similar distinction. PerplexityBot is designed to surface and link websites in search results, while Perplexity-User handles user-requested fetches.

One Google-specific caveat: Google-Extended is a robots.txt control token, not a separate HTTP user-agent string. Google says its content is fetched through existing Google crawlers, while Google-Extended controls certain Gemini training and grounding uses. It does not control AI Overviews or AI Mode. Those draw from the regular Google Search index, which Googlebot governs.

Verify more than the user-agent string

User-agent strings can be spoofed. Where operators publish IP information, verify suspicious or high-volume activity against their current ranges. OpenAI (gptbot.json, searchbot.json, chatgpt-user.json), Anthropic (bots.json) and Perplexity (perplexitybot.json) publish IP ranges, and Google documents how to verify its crawlers by reverse DNS. Once you know who is accessing your site, the next question is what those crawlers are actually doing.

Check what crawlers can actually read

A 200 status code doesn’t mean the crawler saw your content. Vercel and MERJ found that OpenAI’s crawlers, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider don’t execute JavaScript. If your key content renders on the client side, these bots receive an empty shell. Gemini (through Googlebot) and Applebot do render JavaScript. To test, fetch a priority page with JavaScript disabled, or run this to confirm the main content is in the raw HTML:

curl -A "OAI-SearchBot" https://yoursite.com/page

How do you analyze AI crawler traffic in server logs?

Your access logs already contain the data you need: user-agent, requested path, timestamp, status code, and response information. For a default Nginx or Apache combined access log, start by finding requests from a specific bot.

# Find GPTBot requests
grep 'GPTBot' /var/log/nginx/access.log

# Count GPTBot requests
grep 'GPTBot' /var/log/nginx/access.log | wc -l

# Find requests from multiple AI crawlers
grep -E 'OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' /var/log/nginx/access.log

# Count GPTBot requests in Apache logs
grep 'GPTBot' /var/log/apache2/access.log | wc -l

# Filter Cloudflare Logpush JSON
jq -r 'select(.ClientRequestUserAgent | test("GPTBot"; "i"))' cloudflare.json

Raw request volume is only the starting point. You want to know how those requests behave.

Metrics to include in an AI crawler server-log audit:

For example, this command shows status-code distribution for GPTBot in a standard combined log:

grep 'GPTBot' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -nr

And this one surfaces the URLs it requests most often:

grep 'GPTBot' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head -20

The interpretation is where the audit becomes useful.

If OAI-SearchBot repeatedly receives a 403 on a high-value article, ChatGPT search may be unable to access that page. OpenAI says sites that opt out of OAI-SearchBot will not be shown as sourced results in ChatGPT search answers, although navigational links can still appear.

A cluster of 429 responses deserves attention too. It can indicate that a CDN, WAF, or application-level rate limit is throttling legitimate retrieval activity.

A sudden spike from PerplexityBot or Perplexity-User can also be worth investigating. Check whether the burst lines up with a topic trending in AI search, and whether the requests come from Perplexity’s published IP ranges.

For recurring analysis, tools such as GoAccess can turn raw access logs into a searchable dashboard. Existing observability platforms including Datadog, Axiom, Splunk, and Cloudflare can also expose user-agent data if those logs are already part of your stack.

How should you configure robots.txt for AI crawlers?

The first decision is which kinds of access you want to allow.

OpenAI provides one of the clearest examples of why a blanket AI-bot policy can create problems. GPTBot and OAI-SearchBot have independent controls. A company can restrict GPTBot because it does not want content used for training while still allowing OAI-SearchBot so its pages remain eligible for ChatGPT search results.

A basic policy could look like this:

# Block OpenAI training crawler
User-agent: GPTBot
Disallow: /

# Allow ChatGPT search crawler
User-agent: OAI-SearchBot
Allow: /

# Block Anthropic training crawler
User-agent: ClaudeBot
Disallow: /

# Allow Anthropic search crawler
User-agent: Claude-SearchBot
Allow: /

# Allow Perplexity search crawler
User-agent: PerplexityBot
Allow: /

# Control certain Google Gemini training and grounding uses
# Blocking can affect eligible Gemini uses
User-agent: Google-Extended
Disallow: /

Your policy will depend on your organization’s requirements, but the configuration should reflect what each bot actually does.

Common AI crawler access decisions:

Google explicitly states that Google-Extended does not determine whether a site appears in Google Search and is not a Google Search ranking signal.

Check your WAF and CDN too

A correct robots.txt file does not guarantee successful access.

Firewall rules, bot-management settings, CDN configuration, authentication, and rate limits can still produce 403 or 429 responses. The access log tells you whether those controls are working as intended.

Use crawl-delay selectively

Crawl-delay is a non-standard directive, so support varies by operator.

Anthropic explicitly states that its bots support the directive and provides ClaudeBot configuration guidance. Google states that it ignores crawl-delay, and OpenAI doesn’t document support for it, so don’t rely on it to control every bot.

For load management, edge caching and rate-limit rules designed around verified bot traffic can give teams more predictable control.

What is the difference between training and retrieval crawlers?

Training crawlers collect content that may contribute to future model development. Retrieval and search crawlers help AI products find or access information for search experiences and user requests.

The distinction changes what blocking a crawler actually accomplishes.

OpenAI makes the separation explicit: allowing OAI-SearchBot does not require allowing GPTBot.

Anthropic likewise gives site owners separate controls for ClaudeBot, Claude-SearchBot, and Claude-User.

User-triggered fetchers require another layer of consideration. OpenAI says robots.txt rules may not apply to ChatGPT-User because the action originates with a user. Perplexity says the same of Perplexity-User, which ‘generally ignores robots.txt rules.’ Anthropic is the exception: it says blocking Claude-User in robots.txt prevents user-directed retrieval, though that may reduce your visibility in Claude’s web search.

So an audit should avoid collapsing every AI request into a single “AI bot” bucket. A 403 from GPTBot and a 403 from OAI-SearchBot indicate two very different policy outcomes.

How do you measure the impact of AI crawler traffic?

Crawler activity becomes useful when you connect infrastructure data to what users eventually see in AI search.

Use three measurement layers.

1. Monitor crawl health

Start with the server logs.

For retrieval and search crawlers, watch the percentage of requests returning 200, especially on the pages you most want AI engines to surface. Also track 403, 404, and 429 responses, revisit frequency, and whether crawl activity is reaching priority sections of the site.

The goal is to establish a baseline for normal crawler behavior so changes stand out quickly. If OAI-SearchBot suddenly starts receiving 403 responses on pages that previously returned 200, you have a technical problem to investigate before waiting for AI visibility to decline.

Training crawlers can be monitored separately for cadence and coverage. Your own logs will show how often each training crawler revisits priority pages. That baseline is more useful than any industry average.

2. Track citation presence

Crawl access tells you whether an AI system can reach your content. It does not tell you whether the content is actually being selected in answers.

Track citation presence across the prompts and topics your buyers care about. Look at citation rate by engine, brand presence, and competitive share of voice, then compare those changes with crawler behavior.

An AI visibility platform such as Jasper GEO Hub can provide that measurement layer across AI engines. The useful connection is between infrastructure health and the answer surface: if crawler access improves on priority pages, you can watch whether citation visibility changes afterward.

Avoid treating the relationship as automatic. A crawler can access a page perfectly and still choose another source because your page lacks the relevance, freshness, structure, or authority needed for the query.

3. Monitor referral traffic

The final layer is what happens when AI visibility produces a click.

In GA4, start with the AI Assistant channel Google added in May 2026, then use a custom channel group to capture sources it misses, such as Perplexity (Our attribution guide covers the setup). Track those sessions against the landing pages appearing in your crawler and citation data.

Referral traffic should be treated as a directional signal because AI-originated visits are not always cleanly attributable in analytics. The useful question is whether identifiable AI referrals are changing alongside crawl access and citation visibility.

Reading all three layers together gives you a fuller operating view:

Crawl health → citation presence → referral activity

If crawl health deteriorates first and citation presence falls later, you have a technical lead indicator worth investigating. If crawl access remains healthy while citations decline, the problem likely sits elsewhere in your GEO strategy.

How often should you audit AI crawler traffic?

Teams should aim for a recurring cadence to audit AI crawler traffic:

• Weekly: Review AI crawler hit counts, status-code distribution, most-crawled URLs, and unusual changes in crawl behavior.
• Weekly: Spot-check citation presence across your highest-priority branded and non-branded prompts.
• Monthly: Review identifiable AI referral traffic and the landing pages receiving it.
• Monthly: Compare crawl-health trends with citation visibility to identify patterns worth investigating.

The longer you maintain the dataset, the more useful those comparisons become. You can establish what normal crawl activity looks like for your site, identify infrastructure changes quickly, and see whether crawler disruptions precede changes in AI visibility.

Correlation still needs to be interpreted carefully. A crawler visit does not cause a citation, and successful access does not guarantee inclusion in an AI response.

What it gives you is a measurable infrastructure layer underneath your GEO program.

Turn AI crawler activity into a measurable signal

AI crawler traffic no longer has to be an invisible part of your infrastructure.

Your server logs can show which AI systems are accessing the site, where they are going, and whether your technical configuration is helping or preventing them from reaching priority content. Separating training crawlers from retrieval crawlers then gives you a more precise way to decide which access you want to allow.

The next step is connecting that infrastructure data to actual visibility.

When crawler health, citation presence, and referral traffic are tracked together, you can start identifying where problems enter the system and where an optimization is producing a measurable change.

Ready to see your AI visibility score? Discover how the Jasper GEO Hub and Agent give you the analysis and execution guidance you need to optimize AI search visibility.

Frequently asked questions

How do I identify GPTBot in server logs?

Filter your server access logs for the GPTBot user-agent token. For example:

grep 'GPTBot' /var/log/nginx/access.log

You can then group those requests by status code, URL, timestamp, or other log fields to analyze crawl behavior. Because user-agents can be spoofed, verify high-volume activity against OpenAI’s published IP list.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot and OAI-SearchBot perform different functions for OpenAI. GPTBot crawls content that may be used to improve generative AI models, while OAI-SearchBot supports the discovery of websites for ChatGPT search. Site owners can control them independently in robots.txt.

Can I block GPTBot and still appear in ChatGPT search?

Yes. OpenAI allows sites to disallow GPTBot while allowing OAI-SearchBot. OpenAI specifically documents these as independent controls.

Does Google-Extended affect Google Search rankings?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It is a robots.txt control token for certain Gemini training and grounding uses. It also doesn’t control AI Overviews or AI Mode. Those use the standard Google Search index.

What does ClaudeBot crawl?

Anthropic describes ClaudeBot as the crawler used to collect web content that could contribute to model training. Anthropic separately uses Claude-SearchBot for search and Claude-User for user-directed retrieval.

Should I allow every AI crawler?

That depends on what you want each operator to do with your content. Evaluate training access separately from search and retrieval access, then configure each crawler according to your organization’s policy and visibility goals.

Why isn't my site being cited in ChatGPT?

Crawler access is one place to check. Confirm that OAI-SearchBot is allowed in robots.txt and is receiving successful responses on the relevant pages. OpenAI says blocking OAI-SearchBot prevents pages from appearing as sourced ChatGPT search results. If crawl access is healthy, investigate content relevance, freshness, structure, and authority next. Also confirm that key content is in the raw HTML, since OpenAI’s crawlers don’t render JavaScript.

Do AI crawlers render JavaScript?

Most don’t. Research from Vercel and MERJ found that OpenAI, Anthropic, Perplexity and Meta crawlers fetch JavaScript files but don’t execute them. Gemini (through Googlebot) and Applebot do render JavaScript.