robots.txt for AI Crawlers: What to Allow and Block (2026)

When a buyer asks ChatGPT, Perplexity or Claude to recommend a supplier, the answer is built from pages that the assistant's crawlers were allowed to fetch. A single line in your robots.txt can quietly take your business out of that answer. Many sites have also blocked AI crawlers without meaning to, through a blanket rule written years ago or a bot-protection setting switched on at the CDN.

This guide explains which AI crawlers exist, what each one is for, how robots.txt decides who gets in, and how to choose a policy, using each operator's own documentation rather than guesswork. You can check your own site in a few seconds with our free AI Crawler Checker.

Three Kinds of AI Crawler, Three Different Decisions

"AI crawler" is not one thing. The major operators now separate their bots by purpose, and the purpose decides whether blocking one helps or hurts you.

• Training crawlers collect public content that may be used to train future models. Examples: OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot and Meta's Meta-ExternalAgent.
• Search crawlers index pages so an assistant can show them as sources when it answers. Examples: OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot.
• User-request fetchers visit a page because a person asked the assistant about it or shared the link. Examples: ChatGPT-User, Claude-User and Perplexity-User.

Two more names are control tokens rather than crawlers. Google-Extended and Applebot-Extended are not separate bots: Googlebot and Applebot still do the fetching, and the token only tells Google or Apple whether that content may also be used for their AI models.

What Each Operator Says Its Bots Do

OpenAI

OpenAI's crawler documentation describes three agents, each controlled separately. OAI-SearchBot is used to surface websites in ChatGPT's search features, and OpenAI says sites that opt out will not be shown in ChatGPT search answers. GPTBot crawls content that may be used to train its foundation models. ChatGPT-User handles visits initiated by users, and OpenAI notes that because these actions are initiated by a user, robots.txt rules may not apply. OpenAI states that each setting is independent, so a site can allow OAI-SearchBot while disallowing GPTBot, and that search-related robots.txt changes can take about 24 hours to take effect.

Anthropic

Anthropic's help centre lists ClaudeBot (training), Claude-SearchBot (improving search result quality) and Claude-User (fetching a page when a user asks Claude to). It warns that blocking Claude-SearchBot or Claude-User may reduce your visibility in user-directed search, and it asks site owners to opt out through robots.txt rather than by blocking IP addresses, which it says may not work reliably.

Perplexity

Perplexity says PerplexityBot is designed to surface and link websites in its search results and is not used to crawl content for AI foundation models. Its user-triggered fetcher, Perplexity-User, is different: Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.

Google

Google's documentation says Google-Extended controls whether content is used for AI training and grounding in products such as Gemini apps and Vertex AI, and that it does not affect a site's inclusion in Google Search or act as a ranking signal. AI Overviews and AI Mode are part of Search. Google says a page has to be indexed and eligible to show with a snippet to appear in them, and that the controls are the normal Search ones (nosnippet, max-snippet, noindex), which also affect your regular results. Blocking Google-Extended therefore does not remove you from AI Overviews, and blocking Googlebot to stay out of AI Overviews would remove you from Google Search altogether.

How robots.txt Decides Who Gets In

robots.txt was standardised by the IETF in 2022 as RFC 9309. Most surprises come from three of its rules:

1. A crawler follows only one group. If there is a group for its own name, it obeys that group and ignores the User-agent: * group completely. If there is none, it obeys the * group.
2. The longest matching rule wins. Allow: /blog/ beats Disallow: / for a page under /blog/, because it is the more specific match.
3. An empty Disallow: allows everything. It is not the same as Disallow: /, which blocks the whole site.

The first rule causes the classic accident. A site writes User-agent: * with Disallow: / on a staging server, the file is copied to production, and every crawler without its own group (including every AI search bot) is shut out. The opposite also happens: a site adds a User-agent: GPTBot group to block training, and forgets that GPTBot now ignores the useful exceptions listed under *.

Choosing a Policy

There is no single right answer. The choice depends on whether your content is the product or the shop window.

If you want AI assistants to recommend you

Most service businesses, manufacturers, SaaS companies and local businesses are in this group. Allow the search crawlers and user-request fetchers. Decide separately about training crawlers. Allowing them costs little for a business site, but it is a legitimate choice to refuse.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

This example blocks OpenAI's training crawler while keeping the site eligible for ChatGPT search. Anything without its own group follows the last group. Note the trap from the previous section: GPTBot ignores the * group here, which is fine because it is blocked anyway.

If your content is what you sell

Publishers, paid research and course providers often block training crawlers and sometimes search crawlers too. Be aware of the trade-off: an assistant that cannot read your pages cannot cite or link them either. Robots.txt is a request, so content behind a paywall or login is protected by that login, not by robots.txt.

What robots.txt cannot do

• It cannot stop a bot that ignores it, and user-triggered fetchers may legitimately not apply it, as OpenAI and Perplexity both note.
• It cannot undo a firewall block. Several CDN and hosting platforms offer one-click AI-bot blocking. If that is on, your robots.txt "Allow" lines do nothing.
• It does not remove content already collected. Changes apply to future crawling.

Check the Firewall, Not Just the File

Allowing a crawler in robots.txt is half the job. The other half is making sure your server actually lets it in. Look at your web server or CDN logs for the user-agent names above and check the status codes they receive. A crawler that gets 403 or 429 responses is blocked whatever robots.txt says. If you use a CDN's bot management, check its AI-crawler settings and any "verified bot" exceptions.

On our own site, the access logs for a recent four-day window showed OAI-SearchBot, ClaudeBot, PerplexityBot and ChatGPT-User all fetching pages successfully. That is the evidence that matters. A robots.txt that allows a crawler while a firewall blocks it looks fine in an audit and fails in practice.

Where llms.txt Fits

llms.txt is a proposal, published at llmstxt.org in 2024, for a Markdown file at your site root. It contains a one-paragraph summary of the site and curated links to its most useful pages, written for language models rather than search engines. It is easy to create with our llms.txt generator, and you can see a working one at brainguru.in/llms.txt.

Keep expectations realistic. Google says you do not need any new machine-readable files, AI text files or special markup to appear in its AI features, and the other major AI providers have not confirmed that they read llms.txt. It is cheap groundwork that may help some tools understand your site. It does not replace crawlable pages with clear answers, and it is not a ranking factor anyone has confirmed.

Access Is Step One, Not the Whole Job

Being readable makes you eligible to be cited. It does not make you the answer. Assistants tend to cite pages that answer a question directly, state facts consistently and are backed by other sources. In practice that means:

• Pages that answer the questions buyers actually ask, with the answer in the opening sentence.
• Consistent business facts: the same name, location and services on your site, Google Business Profile and directories.
• Structured data that matches the visible page. Our FAQ Schema Generator helps with question-and-answer content, and our guide to schema markup for Indian businesses covers the rest.
• Mentions and reviews on independent sites that assistants already trust.

For how this relates to Google specifically, see our guide to Google AI Overviews for Indian businesses.

A 15-Minute AI Access Audit

1. Run your domain through the AI Crawler Checker and note any crawler marked blocked, and whether it used its own group or the * group.
2. Decide your policy for training crawlers and search crawlers separately, and write it down.
3. Edit robots.txt to match. Keep groups simple, and test again.
4. Check your CDN or firewall for AI-bot blocking, and your server logs for the crawlers' status codes.
5. Publish an llms.txt if you want to, and make sure your key pages answer the questions buyers ask.

Frequently Asked Questions

Will blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, and says each setting is independent. You can block GPTBot and still allow OAI-SearchBot so your pages can appear as sources in ChatGPT search answers.

Does blocking Google-Extended keep my content out of AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. Keeping content out of AI Overviews needs Search controls such as nosnippet or noindex, which also affect your normal search results.

Do all AI crawlers obey robots.txt?

The automatic crawlers named in this guide are documented by their operators as following robots.txt. Fetchers acting on one user's request are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Unknown or disguised bots may ignore it entirely, so only server or CDN rules can enforce a block.

Should a small business allow AI training crawlers?

It is a business choice rather than a technical one. For a typical service business whose pages describe what it sells, allowing training crawlers costs little. If your content is the product, such as paid research or courses, blocking them is reasonable. Either way, allow the search crawlers if you want to be recommended in AI answers.

Is llms.txt required to appear in AI answers?

No. Google says no special AI text files or markup are needed for its AI features, and other providers have not confirmed that they read llms.txt. It is a low-cost proposal that may help some tools, not a requirement or a confirmed ranking factor.

How quickly do robots.txt changes take effect?

It depends on when each crawler next fetches your robots.txt. OpenAI says search-related changes can take about 24 hours to take effect. Plan for at least a day, and check your server logs to confirm the crawler's behaviour has changed.

Where Brainguru Can Help

We help businesses become visible and citable in AI assistants and Google's AI features through our generative engine optimisation (GEO) services, built on technical foundations from our SEO services and local SEO work. Start with the free AI Crawler Checker, or get in touch on +91-8010010000.

Sources: OpenAI, "Overview of OpenAI Crawlers"; Anthropic Help Center, Claude's crawlers; Perplexity, "Perplexity Crawlers"; Google Search Central, "Google's common crawlers" and "AI features and your website"; RFC 9309, Robots Exclusion Protocol; llmstxt.org. Checked October 2026.