Free tools

AI Crawler Robots.txt Tester

Check whether AI crawlers like GPTBot, ClaudeBot, PerplexityBot, Googlebot, and other bots can access your website for AI search visibility.

Copy-paste outputs

The autonomous growth engine for your business.

Brand Armor AI finds where your customers are, decides what to create, distributes it across the channels that matter, and learns from every result.

Discover customer opportunitiesPlan campaignsCreate posts, images, and videosPublish across channelsMeasure what worksAI search visibilityDiscover customer opportunitiesPlan campaignsCreate posts, images, and videosPublish across channelsMeasure what worksAI search visibility
Social publishingBlog and CMS deliveryEmail outreachAutopilot workflowsBrand IdentityWeekly reportsSocial publishingBlog and CMS deliveryEmail outreachAutopilot workflowsBrand IdentityWeekly reports
Shopping IntelligenceCompetitor AnalysisBrand IdentityShopping IntelligenceCompetitor AnalysisBrand Identity

Tool 01

Robots.txt Tester

Test your robots.txt file to see if specific URLs are allowed or blocked. Parse rules, test URLs, and verify crawler access for any user-agent.

Robots.txt Tester
Paste your robots.txt content and test if specific URLs are allowed or blocked.
Test Results
View parsed robots.txt rules and test results for your URLs.

Enter robots.txt content to see parsed rules

How it works

AI Crawler Robots.txt Tester: methodology and worked example

How this tool computes its result

parseRobotsTxt splits pasted text into User-agent blocks, collecting each block's Allow/Disallow/Crawl-delay lines. testUrlAgainstRules finds the first rule whose user-agent matches the requested one (or falls back to "*"), then checks that rule's Allow paths first — if any Allow path matches (via `path.startsWith(allowPath.replace("*",""))`, or literally "/") it returns allowed immediately — and only checks Disallow paths if no Allow rule matched. If neither list matches, the URL is allowed by default.

Worked example

robots.txt = "User-agent: *\nDisallow: /admin/\nAllow: /", tested against URL "https://example.com/admin/settings" with user-agent "*": the matched rule has allow: ["/"] and disallow: ["/admin/"]. Because Allow paths are checked before Disallow paths, and "/" matches every path immediately, the tool reports the URL as ALLOWED with reason "Allowed by rule: Allow: /" — even though intuitively the more specific `/admin/` Disallow should take precedence.

When not to use this tool

The matching algorithm checks all of a rule's Allow entries before any of its Disallow entries, regardless of path specificity or declaration order. This diverges from the official Robots Exclusion Protocol, where the longest (most specific) matching rule wins. For robots.txt files with overlapping Allow/Disallow rules of different specificity — like the worked example above — this tool's verdict can disagree with how Googlebot, GPTBot, or other real crawlers actually interpret the same file.

Common mistakes

  • - Assuming this tool replicates Google's or OpenAI's official longest-match-wins precedence — it instead evaluates every Allow rule before any Disallow rule, in array order, not by specificity.
  • - Leaving User-Agent as the default "*" when the real concern is a named bot (e.g. "GPTBot") that has its own more specific block in the pasted robots.txt.
  • - Relying on wildcard "*" handling in path values — matching is done with a crude `.replace('*', '')` plus `startsWith`, not proper glob or robots.txt wildcard semantics.

Ready to put your growth work on autopilot?

Connect your business once, then let Brand Armor AI find opportunities, create content, publish it, and learn from what brings customers in.

Frequently Asked Questions