How to Configure Robots.txt and XML Sitemaps Safely

Interpret the tool output and prioritize technical SEO work while keeping essential operating instructions on the tool page.

Use the Robots & Sitemap Tool →

What Are robots.txt and sitemap.xml?

Two files every website needs but many site owners get wrong: robots.txt tells search engine crawlers which parts of your site to crawl and which to skip. sitemap.xml gives Google a complete list of your pages, helping crawlers discover and index your content faster.

A missing or misconfigured robots.txt can accidentally block Google from crawling important pages — I've seen sites lose 40% of their indexed pages overnight because a developer added a single Disallow: / line that blocked everything. A missing sitemap means Google has to discover your pages by following links — which can delay indexing by weeks for new content.

How to Use This Robots & Sitemap Generator

For robots.txt: select which bots to allow or block, specify any folders or file types you want excluded (like admin pages, duplicate URL parameters, or staging paths), and click Generate. The tool outputs a ready-to-upload robots.txt file. Save it as robots.txt and upload it to your site's root directory so it's accessible at yourdomain.com/robots.txt.

For sitemap.xml: enter your page URLs (one per line), set the priority and change frequency for each, and click Generate. Upload the resulting file to your site root, then submit the URL to Google Search Console under Indexing → Sitemaps. Google will crawl it and use it to discover your pages.

Common robots.txt Mistakes to Avoid

The three most common robots.txt errors: blocking CSS and JavaScript files (which stops Google from rendering your pages correctly), using Disallow: / for the wrong user-agent, and forgetting to include your sitemap URL at the bottom of the file. Always test your robots.txt using Google Search Console's robots.txt tester after making changes — a syntax error in this file can have site-wide indexing consequences.

For most small sites, the robots.txt should be simple: allow all crawlers, disallow any admin or login paths, and include a pointer to your sitemap. Complexity in robots.txt is usually a sign of a deeper crawl budget or URL structure problem that should be fixed at the source.

Use the Robots & Sitemap Tool →