Robots.txt Tester
Paste robots.txt rules, enter a path, and see whether a crawler should be allowed or blocked by the longest matching rule.
Check crawler rules before publishing
A single misplaced line in robots.txt can quietly deindex an entire section of a site, or leave a staging area you
meant to block wide open to crawlers. The rules are deceptively fiddly: Allow and
Disallow interact, wildcards and user-agent groups add nuance, and the winner is the longest matching
path — not the first one listed. This tester lets you paste your rules, pick a crawler, and enter a path to see
exactly how that combination resolves.
It mirrors the key behavior real crawlers use: it reads the rules that apply to the chosen user-agent (plus the
wildcard * group) and returns the verdict from the longest matching Allow or Disallow. That makes it
ideal for confirming a new rule does what you intend before you push it live — where a mistake is expensive and
slow to notice.
Private and instant
Nothing is fetched — you paste the rules you want to test, and the check runs entirely in your browser.
Frequently Asked Questions
Does this fetch my live robots.txt?
No. Paste the rules you want to test. This keeps the tool browser-only and avoids network calls.
How is the result calculated?
It checks rules for the selected user-agent and wildcard sections, then uses the longest matching Allow or Disallow path.
Is this a full search engine simulator?
No. It is a practical tester for common Allow and Disallow rules.
Need metadata, content structure, and technical SEO cleaned up across a full site? I can help.