Free tools

Free AI Crawler Robots.txt Generator

Build a robots.txt that guides GPTBot, Claude, and Perplexity while keeping human search indexed.

Copy-paste outputs

The autonomous growth engine for your business.

Brand Armor AI finds where your customers are, decides what to create, distributes it across the channels that matter, and learns from every result.

Discover customer opportunitiesPlan campaignsCreate posts, images, and videosPublish across channelsMeasure what worksAI search visibilityDiscover customer opportunitiesPlan campaignsCreate posts, images, and videosPublish across channelsMeasure what worksAI search visibility
Social publishingBlog and CMS deliveryEmail outreachAutopilot workflowsBrand IdentityWeekly reportsSocial publishingBlog and CMS deliveryEmail outreachAutopilot workflowsBrand IdentityWeekly reports
Shopping IntelligenceCompetitor AnalysisBrand IdentityShopping IntelligenceCompetitor AnalysisBrand Identity

Tool 01

AI Crawler Robots.txt Builder

Control AI crawler access without guesswork. Build a robots.txt that keeps humans and assistants aligned.

Policies
Set default access and override by crawler.
robots.txt output
Copy and paste into your site root.
robots.txt
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: CCBot
Allow: /

User-agent: Anthropic-AI
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: xAI-Bot
Allow: /

User-agent: Mistral-Bot
Allow: /

How it works

AI Crawler Robots.txt Builder: methodology and worked example

How this tool computes its result

Builds a robots.txt file from a default policy (Allow: / or Disallow: /) applied to `User-agent: *`, followed by one block per crawler in a fixed list of nine named AI bots (GPTBot, ChatGPT-User, CCBot, Anthropic-AI, Claude-Web, PerplexityBot, Google-Extended, xAI-Bot, Mistral-Bot), each independently toggleable between Allow and Disallow. An optional Crawl-delay value, if set, is appended identically to every single block including the wildcard block. An optional Sitemap URL is appended as the final line.

Worked example

Default policy "Allow", PerplexityBot individually switched to "Disallow", crawl delay "1", sitemap URL "https://example.com/sitemap.xml": output includes `User-agent: *` / `Allow: /` / `Crawl-delay: 1` at the top, a `User-agent: PerplexityBot` / `Disallow: /` / `Crawl-delay: 1` block among the others (all still Allow), and a trailing `Sitemap: https://example.com/sitemap.xml` line.

When not to use this tool

Every rule generated is whole-site (`Allow: /` or `Disallow: /`) — there is no way to block or allow a specific path like `/admin/` per bot, only all-or-nothing access per user-agent. The nine-bot list is also fixed in the component's source and will not automatically include newer AI crawler user-agent strings introduced after this list was written.

Common mistakes

  • - Expecting "Disallow" for a given bot to block only part of the site — it always disallows the entire site (`/`) for that user-agent, with no path-level granularity.
  • - Setting a Crawl-delay value expecting it to apply only to a chosen bot — it is written into every single user-agent block generated, including the wildcard `*` block, whether or not that makes sense for that crawler.
  • - Treating the fixed nine-crawler list as exhaustive — new or renamed AI crawler user-agents released after this list was built will not appear and must be added manually to the generated file.

Ready to put your growth work on autopilot?

Connect your business once, then let Brand Armor AI find opportunities, create content, publish it, and learn from what brings customers in.

Frequently Asked Questions