Technical SEO

Robots.txt Generator

Create clear crawling instructions for search engine bots.

Browser processing · No account required
Robots.txt controls cooperative crawling; it is not access control and does not guarantee removal from search. Review every rule before deploying.

Crawling instructions

User-agent group 1

One product token per line, or * for all crawlers.

Optional and nonstandard. Not supported by Google.

Allow, Disallow, Crawl-delay, or # comments. One directive per line.

Optional. One absolute URL per line.

Live robots.txt

Generated from your current rules. Review before publishing.
User-agent: *
Disallow:
WORK WITH CONFIDENCE

Understand your results

For site maintainers preparing explicit crawl rules for known crawler groups.

The live preview serializes your groups in order, with optional sitemap declarations. Identical allow/disallow paths and repeated user-agents are flagged for review. It is a directive generator, not a full matching simulator: crawlers can differ in supported extensions and how they select groups.

How to use this tool

  1. Start with allow all, or deliberately select a blocking preset and replace the example paths.
  2. Enter one path per line, add additional user-agent groups if needed, and supply sitemap URLs.
  3. Resolve formatting errors, review broad blocking rules, then copy or download robots.txt for the root of the correct host.

What the observations mean

User-agent selects a crawler group. Allow and Disallow describe crawl paths; Sitemap declares an absolute sitemap location. Crawl-delay is an extension with inconsistent support and is not supported by Google.

Worked example

Prepare a crawler group that excludes an example internal search path.

User-agent: *
Disallow: /search/
Sitemap: https://example.com/sitemap.xml

With * as the user-agent, /search/ as a disallowed path and the sitemap URL entered, the generated file declares those directives. Deploy it at the intended host’s /robots.txt after reviewing the actual paths.

Limits and processing

Generation does not deploy a file or simulate every crawler’s path matching. A crawl restriction does not provide authentication or guarantee removal from search. This tool processes form values locally.

Common problems and practical fixes

Accidental site-wide block
Review Disallow: / and test representative public paths before deployment.
Private files listed
Use real authentication. robots.txt is public and does not prevent access or reliably remove known URLs from search.
Wrong host
Deploy the file at /robots.txt on each intended host; subdomains need their own configuration.

Best practices

Keep assets needed for rendering public pages crawlable. Use the remote tester to inspect served directives, while testing crawler-specific matching separately.

Practical tips

  • Disallow: / is a site-wide crawl block for the matching group. More specific Allow paths may override it.
  • Robots.txt is public and is not authentication. Never rely on it to protect confidential files.
  • Crawl-delay is nonstandard and not supported by Google. Keep critical rendering resources crawlable.
A LITTLE MORE CLARITY

Frequently asked questions

Practical answers for using the Robots.txt Generator.

Does Disallow remove a URL from search?

No. A known URL can appear without being crawled. Use appropriate indexing directives and access controls for their separate purposes.

Do the directory presets change my site?

No. Presets only populate editable text. Nothing is uploaded or deployed.

Can I add custom directives?

Custom lines accept Allow, Disallow, Crawl-delay, and comments. Add User-agent and Sitemap through their dedicated controls to keep group boundaries explicit.