robots.txt & .htaccess Generator
Build a robots.txt file or common .htaccess rules from a simple form.
About the robots.txt & .htaccess Generator
Create a robots.txt file or common .htaccess rules from a form. For robots.txt, choose the user agent, the paths to disallow, an optional crawl delay and your sitemap address. For .htaccess on Apache servers, tick rules such as force HTTPS, redirect www to non-www (or the reverse) and add an X-Robots-Tag header, then download the file.
How to use it
- Choose robots.txt or .htaccess.
- Fill in the form: user agent, disallowed paths and sitemap, or the redirect rules you need.
- Check the generated file.
- Download it and upload it to the root of your website.
Tip: robots.txt controls crawling, not indexing. To keep a page out of search results, use a noindex tag or an X-Robots-Tag header instead of blocking it.
Questions
Where does robots.txt go?
In the root of your site, so it opens at https://yourdomain.com/robots.txt.
Does Disallow remove a page from Google?
No. It stops crawling, but a blocked page can still be listed if other sites link to it. Use noindex to keep it out.
Should I add my sitemap?
Yes. A Sitemap line helps search engines find all your important pages.
What is .htaccess?
A configuration file for Apache web servers. It can force HTTPS, redirect www and set headers. Other servers (Nginx, Vercel, Netlify) use their own settings.
Is anything sent to a server?
No. The files are generated in your browser.