Ask four separate questions
Discovery asks how a crawler learns a URL exists. Crawl access asks whether it can request that URL. Indexing asks whether the search engine stores and selects the content for search. Access control asks who is allowed to retrieve the content at all. A sitemap can help discovery and robots.txt can express crawl preferences, but neither guarantees indexing or protects confidential files.
For a new article missing from search, inspect its internal links, final HTTP response, canonical, and indexing directives before adding broader crawl rules. A sitemap containing a URL that redirects elsewhere creates avoidable inconsistency. Prefer a sitemap of final canonical pages that should be indexed, and link to them through the site’s ordinary navigation.
Write robots.txt for the correct host
Place robots.txt at the root of the host it controls. Rules on example.com do not automatically configure a separate subdomain. A User-agent line selects a crawler group; Allow and Disallow express path rules within it. An empty Disallow does not block all crawling, while Disallow: / is a broad block for its matching group. Make broad rules an explicit decision.
The example blocks a search path while keeping the rest of the site crawlable. It does not remove already known search URLs from an index and does not stop a visitor opening them. Avoid listing secret paths as a protection mechanism: the file itself is public. Keep CSS, scripts, and images needed to understand public content available to supporting crawlers.
Sitemap is an absolute URL declaration rather than a group-specific path rule. Crawl-delay is not universally supported; Google does not support that directive. Do not use it as a guaranteed request-rate limiter. The generator prepares text and the remote tester reports directives; neither simulates every crawler’s group selection and path matching.
User-agent: *
Disallow: /search/
Sitemap: https://example.com/sitemap.xmlChoose urlset or sitemapindex
A standard XML urlset lists page URLs, each inside a url element with a loc child. A sitemapindex lists sitemap files inside sitemap elements. Both use the sitemap 0.9 namespace. Use absolute URLs, escape XML characters such as & as &, and include the intended canonical version of each page. Do not add internal API endpoints, test routes, 404 pages, or deliberately noindex pages.
The protocol limits an individual sitemap to 50,000 URLs and 50 MB uncompressed. Split a larger collection and list the files in an index. WebToolspire’s local validator has a smaller 2 MiB input cap; remote validation has tighter limits. An input-limit error is a tool capacity result, not proof that a protocol-compliant larger sitemap is invalid.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://example.com/guides/tap-repair/</loc></url>
</urlset>Use real dates and fix the reported layer
For a small manually maintained list, generate a urlset from reviewed URLs or a local CSV with url/loc and optional metadata columns. Keep one host per file, correct duplicate or invalid entries before exporting, and entity-escape values such as query-string ampersands. A generator should not invent modification dates or silently present a partial list as complete.
lastmod should reflect a meaningful update to the page, not the instant a build ran. Omit it when you cannot establish an accurate date. Use a supported W3C date or datetime, with a timezone when the format requires one. A valid calendar date does not establish that the page really changed on that day. Optional priority and changefreq values do not compel Google’s crawling behavior.
Distinguish malformed XML from an invalid sitemap structure. A missing closing tag or bare ampersand prevents XML parsing. A valid XML file in an unrelated namespace can still fail sitemap checks. Duplicate locations, missing loc fields, and a sitemap index used as a page list require different fixes. Namespaced image and language extensions have additional rules; accepting an extension is not full schema validation.
Validate a sample first, fix the generator at its source, then validate the final file. Inspect a few listed page URLs for their final status and canonical. The validator does not request every entry or child file, so a structurally valid report cannot establish that all destinations work.
Check the response before blaming the sitemap
A 301 or 308 normally signals a permanent move; 302 and 307 express a temporary redirect. A 404 says the requested resource was not found. A 403 or 429 can mean a crawler or analysis client is being denied or rate-limited. Inspect the actual chain and final destination, and remove unnecessary hops from internal links. Do not replace every missing page with a homepage redirect.
Header checks inspect the target response, which can expose X-Robots-Tag or caching directives absent from HTML. A single Worker-to-target TTFB observation measures the time until response headers from that vantage point; it is not browser LCP, a Core Web Vitals assessment, or a whole-site speed score. Repeat measurements under comparable conditions before drawing conclusions. For international pages, hreflang needs actual language alternatives and reciprocal checks; this release only inspects one source document.
Submit, then investigate observations
Once deployed, fetch robots.txt and sitemap.xml from the preferred HTTPS host. Submit the sitemap in the correct verified Search Console property. Submission is a discovery signal, not an indexing request with a guaranteed outcome. Use page inspection and coverage reports to investigate specific URLs, and compare observed canonical choices with your intended URLs.
Keep a short change record when revising crawl rules. Test representative public, restricted, and newly moved paths. Roll back an accidental site-wide block promptly and verify the served file rather than relying on the local copy.
- Public pages and rendering assets remain crawlable.
- Private resources require authentication regardless of crawl rules.
- Sitemap locations are canonical, indexable routes.
- Dates reflect real changes or are omitted.
- Submission and response checks are recorded separately from indexing results.