Robots.txt Generator
Build a robots.txt file for your website in seconds — allow or block search engines and AI training bots, add your sitemap, then copy or download the file. Free, no sign-up, and everything runs in your browser.
1. Start from a preset
2. Per-bot rules
Choose Allow or Block for each crawler. Search bots (Googlebot, Bingbot) should normally stay allowed so your pages can appear in search results.
3. Custom user-agent
4. Allow / Disallow path rules
Rules apply to the user-agent you pick. Use * for all crawlers. A path like /wp-admin/ blocks that whole folder; /search blocks Blogger search pages.
Crawl-delay directive is not supported by Google — Googlebot ignores it. If a crawler is overloading your server, use that search engine's own crawl-rate settings instead of adding a crawl-delay line here, so this generator does not add one.Your robots.txt
What Is a Robots.txt File?
A robots.txt file is a small text file that sits at the root of your website (for example, yoursite.com/robots.txt) and tells search engines and other crawlers which parts of your site they are allowed to visit. Crawlers that follow the rules read this file before they fetch your pages, so it is the standard way to keep bots out of admin areas, search-result pages, and other sections you do not want crawled.
It is a request, not a lock: well-behaved crawlers respect it, but it does not force anyone to stay out, and it does not hide a page from search results on its own.
How to Use This Generator
- Pick a preset — "Allow all" for a normal site, "WordPress" or "Blogger" for the common rules on those platforms, or "Block AI training bots" to keep AI crawlers out while search engines keep working.
- Adjust the per-bot toggles — allow or block individual bots such as Googlebot, Bingbot, GPTBot, or ClaudeBot. Add any other crawler with the custom user-agent field.
- Add path rules if needed — for example, disallow an admin folder or a search page, and allow a single file inside a blocked folder.
- Paste your sitemap URL so crawlers can find it from the same file.
- Copy or download the result and upload the file as
robots.txtat the root of your site. On Blogger, paste the rules into Settings → Crawlers and indexing → Custom robots.txt instead of uploading a file.
Blogger Example
All of the sites on this blog network run on Blogger, and a Blogger robots.txt is simple: allow everything, block the internal search pages (which are thin duplicate pages), and point crawlers at the sitemap. Here is a real example for a Blogger blog:
User-agent: *
Allow: /
Disallow: /search
Sitemap: https://blindarticled.blogspot.com/sitemap.xml
Blogger creates its sitemap automatically at /sitemap.xml on your blog's address, so you only need to add that one line. If you use a custom domain, use your own domain in the sitemap URL instead of the blogspot address.
Blocking AI Training Bots in 2026
Several AI companies run crawlers that collect web content to train their models. The best-known ones are GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose data feeds many AI models), and Google-Extended (Google's crawler for AI training, separate from Googlebot, which handles Search). Blocking them in robots.txt tells those crawlers not to fetch your pages.
What blocking does: it stops those bots from crawling your site going forward, so your new content is not collected by them through crawling. What it does not do: it does not remove content that was already collected, and it does not stop your pages from appearing in AI answers or search results that are built from other sources. If you want your pages out of Google Search entirely, you need a noindex tag on the page itself — robots.txt alone will not do it. Use the "Block AI training bots" preset above to generate the rules for these four crawlers in one click, and keep Googlebot and Bingbot allowed so normal search still works.
FAQ
- Does robots.txt hide my pages from Google?
- No. Robots.txt only asks crawlers not to fetch a page. If Google already knows the URL from a link elsewhere, it can still show the URL in search results without the page content. To keep a page out of search results, add a
noindexmeta tag to the page itself. - Where do I put the robots.txt file?
- At the root of your domain, so it is reachable at
yourdomain.com/robots.txt. On Blogger there is no file upload for this — paste the generated rules into Settings → Crawlers and indexing → Custom robots.txt and save. - Do I need a robots.txt file at all?
- Not strictly. If you have no file, crawlers simply assume everything is allowed. A file is worth having when you want to block specific folders or bots, or when you want to advertise your sitemap in one place.
- Is my input uploaded anywhere?
- No. This generator runs entirely in your browser. Nothing you type — your domain, your paths, or your bot choices — is sent to any server.
- What is the difference between Googlebot and Google-Extended?
- Googlebot crawls pages for Google Search. Google-Extended is a separate crawler that collects content for Google's AI products and model training. Blocking Google-Extended does not affect your Google Search rankings; blocking Googlebot does.
- How long does it take for bots to respect a new robots.txt?
- Crawlers re-read robots.txt regularly, usually within a day or two, so new rules normally take effect quickly. A bot that already has your pages cached will not delete them just because you blocked it afterwards.
Related Tools
Meta Tag Generator · JavaScript Minifier · CSS Minifier · Case Converter · AIToolsForBloggers home
Privacy note: this tool runs locally in your browser. Nothing you type is uploaded, stored, or shared anywhere.