Two traffic systems, two different meanings
Referral traffic is a person arriving from an AI surface. Crawler traffic is an automated system requesting content from your server. They can occur in the same discovery journey, but they are not the same event and should never be reported as one metric.
Crawler traffic answers
Can AI systems access this page?
Referral traffic answers
Did a person arrive from an AI surface?
Machine-to-server activity
Agent chooses or discovers a URL
Search, training, or user-triggered retrieval logic selects a page.
Server receives the request
Your application, CDN, or edge runtime sees the user agent and requested path.
Website returns a status
A 200, redirect, error, or rate limit becomes part of the technical evidence.
No browser session is required
The crawler may never load analytics JavaScript or create a human session.
Person-to-website activity
AI answer presents a link
The user sees a source, product, business, or page worth opening.
A person chooses to click
The browser navigates from the AI surface to your website.
Analytics records the session
Referral or campaign information is retained when browser and privacy settings allow it.
On-site behavior becomes measurable
Landing-page engagement, signup, trial, or purchase can be analyzed.
Field-by-field comparison
| Dimension | AI crawler traffic | ChatGPT referral traffic |
|---|---|---|
| Actor | Automated AI crawler or retrieval agent | Person using a browser |
| Observed by | Server middleware, CDN, edge, or access logs | Web analytics and first-party event tracking |
| Primary evidence | An agent requested a URL | A visitor reached a URL from ChatGPT |
| Typical fields | Agent, path, method, status, time, network metadata | Landing page, source, session, engagement, conversion |
| JavaScript required | No | Usually, for browser analytics |
| Reveals private prompt | No | No |
| Proves recommendation | No | No; the click may follow many answer types |
| Best use | Coverage, access, technical diagnosis | Demand capture, landing-page performance, conversion |
The missing middle
Between the server request and the human visit sits the actual AI answer: which brand appeared, how it was framed, which competitors were recommended, and which sources were cited. That layer requires recurring prompt monitoring.
Explore prompt monitoringCrawler monitoring
Successful requests by agent, purpose, page, and status
Prompt monitoring
Mention and recommendation share across repeated prompts
Citation analysis
Cited domains, owned URLs, competitor sources, and citation quality
Web analytics
AI-attributed sessions, landing pages, engagement, and conversion
Use crawler middleware or trusted server logs for recognized machine requests. Do not depend on a browser pixel that crawlers may never execute.
Use web analytics for human sessions, landing pages, engagement, and conversion. Expect privacy tools and navigation behavior to affect attribution.
Join the layers by date, platform, topic, and landing page trends. Avoid inventing a user-level key that the source systems do not provide.
Privacy note: neither dataset requires collecting the private AI conversation. Keep crawler tokens server-side, avoid raw cookies and authorization headers, and treat referral attribution as a session-level signal rather than a transcript.
Connect the channels
Brand Armor AI provides server-side crawler activity alongside prompt monitoring, citations, competitors, and content actions. Use your web analytics for the separate referral and conversion layer.
What you can measure
Recognized crawler purpose, provider, requested page, and status
Observed AI answers across recurring buyer prompts
Citation sources and competitor recommendation share
A clear boundary between crawler activity and browser sessions
Frequently asked questions
It is a human browser session that reaches your website from a ChatGPT surface and retains referral or campaign information that an analytics platform can attribute to ChatGPT.
It is an automated HTTP request from a recognized AI crawler or agent to your server. It may support training, search discovery, indexing, or user-triggered retrieval depending on the agent.
Many crawlers request HTML directly and do not execute the browser-side analytics JavaScript used to create sessions. Server-side crawler monitoring and browser analytics observe different systems.
Sometimes an AI system may fetch a page and a person may later click it, but there is no dependable one-to-one relationship. A crawler can fetch without showing a link, and a person can visit from an answer without a crawler request occurring at that moment.
They answer different questions. Crawler traffic matters for access and technical coverage. Referral traffic matters for human visits and on-site outcomes. Prompt results and citations are needed to measure answer visibility between those layers.
Crawler identities and platform behavior change. Use provider documentation as the source of truth and review it before changing robots, firewall, or CDN rules.
Official guidance says publishers can measure ChatGPT referral traffic in analytics and separately control OAI-SearchBot and GPTBot access.
Official information about ChatGPT search, linked citations, and website availability through OAI-SearchBot.
Officially separates automated PerplexityBot discovery from user-triggered Perplexity-User requests.
Related guides
Measurement
Learn what ChatGPT referral data and crawler logs can reveal, what remains private, and how to build a reliable demand picture without inventing attribution.
Read guideCrawler intelligence
Separate crawler access from citations, mentions, recommendations, human visits, and conversions using an evidence-based measurement ladder.
Read guideCrawler intelligence
A practical guide to OpenAI, Anthropic, Perplexity, and Google crawler identities, controls, and the evidence each request provides.
Read guidePrompt strategy
Choose a defensible prompt-set size based on buyer journeys, coverage, model variance, markets, and the decisions your team needs to make.
Read guidePrompt strategy
Use branded, category, comparison, problem, and local-market prompts for the right measurement job instead of blending incompatible signals.
Read guideMeasurement
Understand the separate timelines for crawler access, retrieval, indexing, citations, recommendations, and future model training.
Read guide