AI Crawler Checker & llms.txt Generator
Find out whether ChatGPT, Claude, Perplexity, Gemini and other AI crawlers are allowed to read your site, fix your robots.txt, and create an llms.txt file — in one place.
1. Check which AI crawlers can read your site
2. Generate an llms.txt file
# Your Site
Upload it to your site root so it opens at https://your-domain/llms.txt, served as plain text.
AI Assistants Can Only Recommend Pages They Can Read
When buyers ask an AI assistant for a supplier, the answer is built from pages its crawlers could fetch. A single robots.txt line can quietly remove you from that list.
Search agents
OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages that AI answers cite as sources. Blocking them is the usual reason a brand never appears.
User-request fetchers
ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks about it or shares the link.
Training crawlers
GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens decide whether content may train future models.
Audit Your AI Visibility in 4 Steps
Check your domain
The checker reads robots.txt exactly as a crawler would: the crawler's own group if it has one, otherwise the User-agent: * group, with the longest matching rule winning.
Decide your policy
Most businesses that want leads from AI answers allow the search and user-fetch agents. Decide separately about training crawlers.
Fix robots.txt — and the firewall
Copy the generated lines for any crawler you want to allow. Then check that your CDN or firewall does not block it anyway; many bot-protection settings block AI crawlers by default.
Add llms.txt
Fill in the generator with your summary and key pages, download the file and upload it to your site root. See our own llms.txt for a working example.
Access Is Step One of Generative Engine Optimisation
Being crawlable does not make you recommended. AI answers favour sources that state facts plainly and consistently: who you are, where you operate, what you offer and for whom, with evidence. That means:
- Clear, consistent business details on your site and profiles (name, location, services).
- Pages that answer real buyer questions directly, with the answer in the opening sentence.
- Structured data that matches the visible page — the FAQ Schema Generator helps with Q&A content.
- Mentions and reviews on independent sites that AI systems already trust.
Learn how we approach this in our GEO services.
AI Crawlers and llms.txt: Frequently Asked Questions
Enter your domain above. The checker reads your robots.txt and reports whether OpenAI's three agents are allowed: OAI-SearchBot (which ChatGPT search uses to show your pages as sources), GPTBot (training) and ChatGPT-User (fetches made for a user). A firewall or CDN bot rule can still block them, so confirm with your server logs.
Not necessarily. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, and they are controlled separately. Blocking GPTBot while allowing OAI-SearchBot is a common choice for sites that want to be cited in answers without contributing to training.
No. Google documents Google-Extended as a control for AI training and grounding in Gemini apps and Vertex AI, and says it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search: a page needs to be indexed and eligible for a snippet, and the controls are the normal Search ones such as nosnippet, max-snippet and noindex — which also affect regular results.
A plain-text Markdown file at your site root, proposed at llmstxt.org in 2024, that gives AI systems a short summary of your site and links to its most useful pages. It is a proposal, not a standard: Google says no special AI text files or markup are needed to appear in its AI features, and other AI providers have not confirmed that they read it. Treat it as cheap groundwork, not a ranking lever.
The automatic crawlers listed here are documented by their operators as respecting robots.txt. Fetchers acting on one user's request are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Unknown or disguised bots may ignore it altogether, so server or CDN rules are the only way to enforce a block.
It is a business decision. If you want your brand recommended in ChatGPT, Perplexity, Claude or Gemini answers, the search and user-fetch agents need access. If your content is your product — paid research, for example — blocking training crawlers protects it. Many businesses allow search agents and decide training agents case by case.
Usually a general rule such as "User-agent: *" with "Disallow: /" applies to every crawler without its own group. The Rule used column shows whether a crawler followed its own group or the wildcard group. A robots.txt that returns a server error is also treated as "do not crawl".
Want to Be the Brand AI Recommends?
We help businesses become visible and citable in ChatGPT, Perplexity, Gemini and Google AI Overviews.
More Free Tools From Brainguru
All free, no sign-up, and they run in your browser.