robots.txt Generator: Block Folders, AI Crawlers and Add Your Sitemap

What is it?

The robots.txt Generator builds a correct robots.txt file and lets you Check a path against it.

Who is it for?

Website, Blogger and WordPress owners controlling what search engines and AI crawlers can visit, and developers launching new sites.

How to use it

Choose the Default: Allow all search engines (recommended), or Block everything (site not ready yet).
Start from the Blogger preset, WordPress preset or Shop preset, then edit Block these paths (one per line) and But allow (optional).
Choose rules for AI crawlers (like GPTBot, ClaudeBot or Google-Extended) and add your Sitemap URL(s).
Copy or Download the file.

Where it goes

It must be at the root: https://yoursite.com/robots.txt.
Blogger: Settings → Crawlers and indexing → turn on Enable custom robots.txt → paste.
WordPress: an SEO plugin edits it, or upload the file to the site’s root folder.

Important

robots.txt asks crawlers not to visit; it doesn’t hide pages. Blocked pages can still appear in Google if linked. Use noindex to keep a page out of results.
Never block CSS, JS or image folders your pages need; Google must see the page as visitors do.
Don’t leave “Block everything” on after launch. It’s a common reason a new site gets no traffic.

Worked example (Blogger)

User-agent: *
Disallow: /search
Allow: /
Sitemap: https://yourblog.com/sitemap.xml

This keeps search and label pages (thin, duplicate lists) out of crawling, while every post and page stays open.

AI crawlers

Blocking GPTBot, ClaudeBot or Google-Extended stops those companies’ crawlers from using your pages for AI training, if they respect robots.txt. It doesn’t affect normal Google Search ranking.

Errors and how to fix them

Check a path says blocked when it shouldn’t be: a broader rule like /admin also blocks /administration; add a trailing slash or an Allow rule.
Google still shows a blocked page: robots.txt doesn’t remove pages; use noindex or Search Console’s removal tool.

Open the tool: robots.txt Generator

Still stuck?

Contact support. Tell us which tool or page you were using, what you tapped, what you saw, and your phone or computer and browser (for example: Android phone, Chrome). We reply by email.