Robots.txt Tester
Retrieve a public robots.txt file and review crawler groups, directives, and syntax warnings.
Inspect a public website
Analyze URL sends the submitted URL, including its query parameters, to the WebToolspire API. The API checks public DNS and requests the target without your browser cookies or authorization. Do not submit secret URLs. The target is normalized to /robots.txt on the submitted host.
One URL, an eight-second server deadline, and at most five redirects. Remote robots.txt is limited to 32 KiB.
Understand your results
For site owners reviewing the actual crawl instructions published on a public host.
The API requests /robots.txt at the submitted origin, then lists real user-agent groups, rules, sitemap declarations, and syntax observations. A 404 is reported as a missing file. This parser does not perfectly reproduce any crawler’s matching or group-selection behavior.
How to use this tool
- Enter a public HTTP(S) website URL. Its path is replaced with /robots.txt.
- Analyze the URL to retrieve a bounded plain-text file. If this deployment has no API configured, the interface explains that no request was sent.
- Review the actual contents, line-numbered warnings and groups. Copy or download the retrieved file for your audit.
What the observations mean
The report lists the actual fetched robots groups, directives, and warnings. A 404 is reported as an absent file, rather than an invented ruleset. It does not simulate every crawler’s matching algorithm.
Worked example
Compare an expected crawler group with the file actually served on a host.
Public website: https://example.com/
Possible served directive: Disallow: /search/If the host serves that directive in its robots.txt, the report lists the observed group and path along with response information. This is a conditional illustration, not a result fetched from example.com.
Limits and processing
Requires the configured API and retrieves one bounded public robots.txt response. It reports directives and availability, not a complete crawler-specific allow/disallow simulation or indexing decision.
Common problems and practical fixes
- Rules differ from local file
- Inspect the submitted host and redirect destination, then deploy the correct file.
- Conflicting paths
- Review user-agent group boundaries and the intended crawler’s matching rules.
- Fetch denied
- Check public access and bot protection. An API error does not establish that Googlebot sees the same response.
Best practices
Compare several representative paths with the intended crawler’s documented behavior. Never use crawl rules to protect private resources.
Practical tips
- Robots.txt is a public instruction file, not an access-control mechanism.
- A broad Disallow: / may deliberately prevent crawling for one group; review the matching user-agent.
- Crawl-delay is not supported by Google. Duplicate or conflicting rules need context rather than an automatic penalty label.
Frequently asked questions
Practical answers for using the Robots.txt Tester.
Will this test whether a URL is indexed?
No. It retrieves crawl instructions. Indexing, authentication, and crawling are separate concerns.
What happens if robots.txt returns 404?
The tool reports the actual 404 and unavailable file. It does not invent an empty successful file.
Does it follow every sitemap listed?
No. Sitemap declarations are reported as text; their URLs are not fetched.