Robots.txt Generator With AI Crawler Rules
Create a correct robots.txt in seconds: keep private paths out of crawlers, add your sitemap and decide how GPTBot, ClaudeBot, Perplexity and other AI bots may use your site.
"Block AI training only" keeps ChatGPT search, Claude search and Perplexity able to cite you. Googlebot and Bingbot are never blocked by these options.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /search? Sitemap: https://www.example.com/sitemap.xml
Upload it to your site root so it opens at /robots.txt, then check it with the AI Crawler Checker. robots.txt controls crawling, not indexing: to keep a page out of Google, use a noindex tag and leave the page crawlable.
Three Mistakes That Cost Rankings
A robots.txt file is short, but one wrong line can hide an entire site from search.
A staging file on the live site
"User-agent: *" with "Disallow: /" blocks every crawler. It is meant for test servers and often gets copied to production by mistake.
Blocking CSS and JavaScript
Google renders pages like a browser. Hide the files it needs and it may misread your layout and content.
Using it to hide pages
Blocked pages can still be indexed from links. Use noindex for pages you want out of search results.
Create Your robots.txt in 4 Steps
List what crawlers should skip
Admin areas, carts, checkout, internal search results and filter URLs that create endless combinations.
Choose your AI crawler policy
Allow all, block training only, or block every AI bot. The tool never blocks Googlebot or Bingbot with these options.
Add your sitemap
Use the full https:// address of your XML sitemap so every crawler can find all your pages.
Upload and verify
Upload the file to your site root, then test it with our AI Crawler Checker and Google Search Console's robots.txt report.
Robots.txt: Frequently Asked Questions
A plain-text file at the root of your site (yourdomain.com/robots.txt) that tells crawlers which paths they may fetch. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. Well-behaved crawlers read it before crawling your site.
Not reliably. robots.txt controls crawling, not indexing. Google says a blocked URL can still be indexed without its content if other pages link to it. To keep a page out of search results, allow crawling and add a noindex meta tag or header instead.
It depends on your goal. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot keeps your content out of future model training. Blocking search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot also stops those assistants citing your pages. Most businesses that want leads block training at most and allow search crawlers.
No. Googlebot ignores the Crawl-delay line, and Google stopped supporting noindex rules inside robots.txt in 2019. Use a noindex meta tag for indexing control.
No. Google needs CSS and JavaScript to render your pages the way visitors see them. Blocking them can hurt how your pages are understood and ranked.
In the root of each host, so it opens at https://www.yourdomain.com/robots.txt. A robots.txt on one subdomain does not apply to another. Google reads up to 500 KiB of the file.
Want a Technical SEO Check?
We review crawling, indexing, sitemaps and site speed together, so fixes in one place don't break another.
More Free Tools From Brainguru
All free, no sign-up, and they run in your browser.