Provider-by-provider crawler map

AI answer crawlers vs indexing crawlers vs training bots

“AI bot” is not one category. Providers use different identities for user-triggered retrieval, search discovery, model training, and control policies. Classifying them correctly prevents bad robots rules and misleading analytics.

Answer retrieval

Search/indexing

Training/control

The classification rule

Classify the documented agent first. Then interpret the observed request. Never infer a citation, recommendation, user location, or private prompt from the user agent alone.

Crawler directory

Similar-looking requests can represent different platform jobs

OpenAI

Three identities should be measured independently.

Answer retrieval

ChatGPT-User

A user-triggered action may cause ChatGPT to retrieve a page.

Interpretation/control

Do not interpret it as the private prompt or the final answer.

Search/indexing

OAI-SearchBot

Supports discovery and surfacing of websites in ChatGPT search.

Interpretation/control

Review robots.txt, host, CDN, and published IP access.

Training/control

GPTBot

Collects content that may be used to improve future models.

Interpretation/control

Use the named robots.txt group for training preferences.

Anthropic

Claude separates user actions, search quality, and model development.

Answer retrieval

Claude-User

Supports content access when a Claude user requests it.

Interpretation/control

Blocking can reduce access during user-directed retrieval.

Search/indexing

Claude-SearchBot

Navigates web content to improve search result quality.

Interpretation/control

Blocking can reduce search visibility and answer accuracy.

Training/control

ClaudeBot

Collects web content that may contribute to model training.

Interpretation/control

Anthropic documents robots.txt and Crawl-delay support.

Perplexity

Perplexity separates automated search discovery from user-requested retrieval.

Answer retrieval

Perplexity-User

A user request may cause Perplexity to access a page for an answer.

Interpretation/control

Official documentation notes user-triggered fetch behavior.

Search/indexing

PerplexityBot

Surfaces and links websites in Perplexity search results.

Interpretation/control

Perplexity publishes user-agent and IP information for verification.

Google

Google Search crawling and generative-AI controls do not map to one dedicated Gemini crawler.

Search/indexing

Googlebot

Crawls content for Google Search products and their search index.

Interpretation/control

Use Google crawler verification and Search crawling guidance.

Training/control

Google-Extended

A robots.txt product token controlling certain Gemini training and grounding uses.

Interpretation/control

It is not a separate HTTP user agent and does not control Google Search inclusion.

Policy framework

Decide access by purpose, not fear or hype

A blanket allow or block rule can combine decisions that have different business consequences. Review discovery, live retrieval, and training separately with legal, security, SEO, and content owners.

Do we want AI search discovery?

Evaluate search and indexing agents separately from training. Blocking search-oriented agents can reduce the ability of AI search products to retrieve or surface your pages.

Do we permit model-training collection?

Apply the provider-specific training control based on your content policy. Do not assume this is the same switch as current answer retrieval.

Can agents reach priority pages successfully?

Check status codes, redirects, WAF rules, bot challenges, authentication, and rate limits for the exact routes that matter.

Can we verify the request identity?

Use official verification guidance when available. A matching user-agent string without network verification can be spoofed.

Good practice

Match the documented user-agent, use official network verification when provided, record status codes, and preserve the provider's purpose classification in reporting.

Bad practice

Group every bot as “AI answers,” assume every request represents buyer demand, or maintain permanent firewall rules from an unofficial and outdated list.

Verification is part of classification

User-agent strings are declarations, not cryptographic identity. Some providers publish IP ranges or reverse-DNS guidance; others advise relying on robots controls rather than fixed addresses. Follow each provider's current instructions and monitor requests that fail verification separately.

Match the expected user-agent token.

Apply official IP or DNS verification where available.

Keep provider lists current rather than hard-coding forever.

Record unverified lookalikes as candidates, not confirmed agents.

Avoid collecting cookies, authorization headers, or request bodies.

Build provider-specific crawler guidance

Classify real activity

Separate answer, indexing, training, and other AI requests in one crawler intelligence view

Brand Armor AI recognizes supported crawler identities, groups requests by purpose, and lets teams filter the chart, requested pages, live stream, coverage matrix, network origins, and crawler health.

Start free trial Explore crawler monitoring

What you can measure

Purpose-level filters across the crawler analytics page

Detected agents, requested pages, live requests, and status codes

Provider and page coverage matrix with searchable routes

Server-side collection designed to avoid raw browser data

Questions? Email admin@brandarmor.ai

Frequently asked questions

AI crawler classification FAQ

What is the difference between an AI answer agent and a training bot?

An answer or user-directed agent retrieves content for a current user action or answer workflow. A training crawler collects public web content that may contribute to future model development. Their timing and business meaning are different.

Is OAI-SearchBot the same as GPTBot?

No. OpenAI documents OAI-SearchBot for search discovery and GPTBot for potential model training. They have separate controls and should be classified separately in analytics.

Does Google-Extended have its own HTTP user agent?

No. Google documents Google-Extended as a robots.txt control token rather than a separate request user-agent string. Crawling is performed with existing Google user agents.

Should every AI crawler be allowed?

That depends on your goals, content policy, legal requirements, infrastructure, and provider. Treat search discovery, user-triggered retrieval, and model training as separate policy decisions rather than using one blanket rule.

Can a user agent string be spoofed?

Yes. A user-agent claim alone is not strong identity proof. Where a provider publishes verification methods or IP ranges, combine them with the expected user-agent and maintain current rules.

Primary references

Crawler identities and platform behavior change. Use provider documentation as the source of truth and review it before changing robots, firewall, or CDN rules.

OpenAI: Publishers and Developers FAQ

Official OAI-SearchBot and GPTBot guidance for publishers and website owners.

Anthropic crawler documentation

Official definitions and controls for ClaudeBot, Claude-User, and Claude-SearchBot.

Perplexity crawler documentation

Official definitions, user agents, and IP sources for PerplexityBot and Perplexity-User.

Google common crawlers

Official Googlebot and Google-Extended behavior, including the control-token distinction.

Related guides

Continue the measurement journey

Browse all guides

Measurement

Can you see ChatGPT visitor prompts?

Learn what ChatGPT referral data and crawler logs can reveal, what remains private, and how to build a reliable demand picture without inventing attribution.

Read guide

Crawler intelligence

Does crawler traffic mean a recommendation?

Separate crawler access from citations, mentions, recommendations, human visits, and conversions using an evidence-based measurement ladder.

Read guide

Measurement

ChatGPT referrals vs crawler traffic

Understand the difference between a person arriving from ChatGPT and an AI system requesting a page from your server.

Read guide

Prompt strategy

How many prompts should you track?

Choose a defensible prompt-set size based on buyer journeys, coverage, model variance, markets, and the decisions your team needs to make.

Read guide

Prompt strategy

Branded vs non-branded prompts

Use branded, category, comparison, problem, and local-market prompts for the right measurement job instead of blending incompatible signals.

Read guide

Measurement

How quickly does AI search update?

Understand the separate timelines for crawler access, retrieval, indexing, citations, recommendations, and future model training.

Read guide