AI Crawler Robots.txt Tester

How this tool computes its result

parseRobotsTxt splits pasted text into User-agent blocks, collecting each block's Allow/Disallow/Crawl-delay lines. testUrlAgainstRules finds the first rule whose user-agent matches the requested one (or falls back to "*"), then checks that rule's Allow paths first — if any Allow path matches (via `path.startsWith(allowPath.replace("*",""))`, or literally "/") it returns allowed immediately — and only checks Disallow paths if no Allow rule matched. If neither list matches, the URL is allowed by default.

Worked example

robots.txt = "User-agent: *\nDisallow: /admin/\nAllow: /", tested against URL "https://example.com/admin/settings" with user-agent "*": the matched rule has allow: ["/"] and disallow: ["/admin/"]. Because Allow paths are checked before Disallow paths, and "/" matches every path immediately, the tool reports the URL as ALLOWED with reason "Allowed by rule: Allow: /" — even though intuitively the more specific `/admin/` Disallow should take precedence.

When not to use this tool

The matching algorithm checks all of a rule's Allow entries before any of its Disallow entries, regardless of path specificity or declaration order. This diverges from the official Robots Exclusion Protocol, where the longest (most specific) matching rule wins. For robots.txt files with overlapping Allow/Disallow rules of different specificity — like the worked example above — this tool's verdict can disagree with how Googlebot, GPTBot, or other real crawlers actually interpret the same file.

Common mistakes

• - Assuming this tool replicates Google's or OpenAI's official longest-match-wins precedence — it instead evaluates every Allow rule before any Disallow rule, in array order, not by specificity.
• - Leaving User-Agent as the default "*" when the real concern is a named bot (e.g. "GPTBot") that has its own more specific block in the pasted robots.txt.
• - Relying on wildcard "*" handling in path values — matching is done with a crude `.replace('*', '')` plus `startsWith`, not proper glob or robots.txt wildcard semantics.

Related tools