Robots.txt Generator
Create clear crawling instructions for search engine bots.
Crawling instructions
Optional. One absolute URL per line.
Live robots.txt
User-agent: * Disallow:
Understand your results
For site maintainers preparing explicit crawl rules for known crawler groups.
The live preview serializes your groups in order, with optional sitemap declarations. Identical allow/disallow paths and repeated user-agents are flagged for review. It is a directive generator, not a full matching simulator: crawlers can differ in supported extensions and how they select groups.
How to use this tool
- Start with allow all, or deliberately select a blocking preset and replace the example paths.
- Enter one path per line, add additional user-agent groups if needed, and supply sitemap URLs.
- Resolve formatting errors, review broad blocking rules, then copy or download robots.txt for the root of the correct host.
What the observations mean
User-agent selects a crawler group. Allow and Disallow describe crawl paths; Sitemap declares an absolute sitemap location. Crawl-delay is an extension with inconsistent support and is not supported by Google.
Worked example
Prepare a crawler group that excludes an example internal search path.
User-agent: *
Disallow: /search/
Sitemap: https://example.com/sitemap.xmlWith * as the user-agent, /search/ as a disallowed path and the sitemap URL entered, the generated file declares those directives. Deploy it at the intended host’s /robots.txt after reviewing the actual paths.
Limits and processing
Generation does not deploy a file or simulate every crawler’s path matching. A crawl restriction does not provide authentication or guarantee removal from search. This tool processes form values locally.
Common problems and practical fixes
- Accidental site-wide block
- Review Disallow: / and test representative public paths before deployment.
- Private files listed
- Use real authentication. robots.txt is public and does not prevent access or reliably remove known URLs from search.
- Wrong host
- Deploy the file at /robots.txt on each intended host; subdomains need their own configuration.
Best practices
Keep assets needed for rendering public pages crawlable. Use the remote tester to inspect served directives, while testing crawler-specific matching separately.
Practical tips
- Disallow: / is a site-wide crawl block for the matching group. More specific Allow paths may override it.
- Robots.txt is public and is not authentication. Never rely on it to protect confidential files.
- Crawl-delay is nonstandard and not supported by Google. Keep critical rendering resources crawlable.
Frequently asked questions
Practical answers for using the Robots.txt Generator.
Does Disallow remove a URL from search?
No. A known URL can appear without being crawled. Use appropriate indexing directives and access controls for their separate purposes.
Do the directory presets change my site?
No. Presets only populate editable text. Nothing is uploaded or deployed.
Can I add custom directives?
Custom lines accept Allow, Disallow, Crawl-delay, and comments. Add User-agent and Sitemap through their dedicated controls to keep group boundaries explicit.