Robots.txt Generator
Set your allow and disallow rules, crawl-delay, and sitemap to generate a valid robots.txt file in seconds.
Control what crawlers request
robots.txt is the first file most crawlers fetch, and getting it right keeps them focused on the pages that matter while steering them away from admin panels, cart URLs, and endless filtered parameter pages that waste crawl budget. This generator builds a valid file from simple inputs: the user-agent, the paths to allow and disallow, an optional crawl-delay, and your sitemap URL.
One important caveat — Disallow controls crawling, not indexing. A blocked URL can still appear in results if it's
linked from elsewhere, so to truly keep a page out of Google use a noindex meta tag and leave it crawlable. Copy
the output to a file named robots.txt at your site root.
Private and instant
Generated entirely in your browser — nothing is uploaded.
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain-text file at the root of your site that tells search engine crawlers which paths they may or may not request. It guides crawling but does not enforce security.
Does Disallow keep a page out of Google?
Not reliably. Disallow blocks crawling, but a blocked page can still be indexed if linked elsewhere. To keep a page out of the index, use a noindex meta tag and allow crawling.
Should I list my sitemap in robots.txt?
Yes. Adding a Sitemap line helps crawlers discover your sitemap. This generator includes it automatically when you provide the URL.
Is this tool free and private?
Yes. The file is generated in your browser — nothing is uploaded or stored.
Want your crawlability and indexing handled end to end? Let’s talk.