Robots.txt Generator

robots.txt is a plain-text file at the root of your site (example.com/robots.txt) that tells search engine crawlers which paths they may visit. This generator builds one from simple controls: choose Allow All or Disallow All as the default, list any directories to restrict (one per line), optionally set a crawl-delay, and paste your XML sitemap URL.

The output follows the standard directive format — User-agent, Disallow, Allow, Sitemap — so it works with Google, Bing, and every other mainstream crawler.

Because the syntax is unforgiving (one misplaced slash can deindex a site), generating the file beats hand-typing it.

Most sites have sections that shouldn't be crawled: admin panels, cart and checkout pages, internal search results, staging directories. Letting crawlers wander through them wastes your crawl budget — the number of pages a search engine will fetch from your site per visit — on URLs that will never earn traffic.

The Sitemap line is the constructive half of the file: it points crawlers straight at your canonical list of URLs, which speeds up discovery of new content.

Know the limits, though. robots.txt is a politeness convention, not security — compliant bots obey it, malicious ones read it as a map. And blocking a URL doesn't remove it from Google's index if other sites link to it; for genuine removal, use a noindex meta tag instead.

1. Choose your default access: Allow All for a normal public site.

2. List directories to block, one per line — /admin/, /cart/, /search/ are typical.

3. Add your sitemap URL (e.g., https://example.com/sitemap.xml).

4. Copy the output, save it as robots.txt, and upload it to your site's root directory — then verify it at yourdomain.com/robots.txt.

Settings: Allow all, restrict /private/, sitemap added

Output:

User-agent: * Disallow: /private/ Sitemap: https://example.com/sitemap.xml