Robots.txt Generator — Free Online Tool
Generate a valid robots.txt file to control how search engines crawl your site.
About Robots.txt Generator
The Robots.txt Generator builds a valid robots.txt file that controls how search engine crawlers access your website. Choose whether to allow or block all robots, list any folders you want to keep private, and add your sitemap URL to get clean, correctly formatted output.
A well-structured robots.txt protects sensitive areas and points crawlers to your sitemap for better indexing. The file is generated instantly in your browser, so you can copy it and upload it to your site root in moments.
How to Use Robots.txt Generator
- Choose whether to allow or disallow all robots by default.
- Add any custom Disallow paths, one per line.
- Enter your sitemap URL if you have one.
- Click 'Copy' and save the output as robots.txt in your site's root folder.
Key Features
Allow or disallow all crawlers with one setting
Add unlimited custom Disallow paths
Automatically formats paths with a leading slash
Includes an optional Sitemap directive
Live output that updates as you edit
Free, private, and runs entirely in your browser
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain text file in your site's root that tells search engine crawlers which pages or folders they may or may not access. It is the first file most crawlers request when visiting a site.
Where should I put the robots.txt file?
Place it at the root of your domain, for example https://example.com/robots.txt. Crawlers only look for it at that exact location.
Does Disallow remove a page from Google?
No. Disallow stops crawling but does not guarantee removal from search results. To keep a page out of the index, use a noindex meta tag and allow crawling so Google can read it.
Should I include my sitemap?
Yes. Adding a Sitemap line helps search engines discover all your important URLs faster, which can improve crawl efficiency.