Quick answer: To configure robots.txt for AI crawlers, add a separate User-agent group for each bot (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot) with its own Allow or Disallow rules. Allow the search-focused bots if you want AI citations, block training bots only if your business requires it, and never edit the Googlebot or Bingbot groups.
A few years ago, robots.txt was a boring file that only technical SEOs opened. Today it is one of the most important decisions on your site. A single wrong line can make your brand invisible in ChatGPT, Perplexity and Claude, or it can quietly hand your content to every model trainer on the internet.
The problem is that most advice on this topic falls into two extremes. One camp says block every AI bot to protect your content. The other says allow everything because AI is the future. Neither is right for every site. The correct configuration depends on what you want from AI search: citations, traffic, brand mentions, or protection of premium content.
In this guide, I’ll walk you through exactly how to configure robots.txt for GPTBot, ClaudeBot and PerplexityBot, how to separate training bots from search bots, which rules protect your Google rankings, and how to test everything before you publish. If you are new to this space, you may also want to read my overview of AEO and GEO first, because robots.txt is only one layer of AI crawlability.
What Does robots.txt Actually Do?
Robots.txt is a plain text file placed at the root of your domain (for example, https://yourdomain.com/robots.txt). It tells crawlers which URLs they may or may not request. The format is standardized as the Robots Exclusion Protocol (RFC 9309), and Google’s robots.txt documentation explains how Googlebot reads it.
Before you edit anything, understand three limits:
- It is a request, not a lock. Reputable crawlers honor it voluntarily. Badly behaved scrapers ignore it. If content must be private, use authentication, not robots.txt.
- It controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of an index, use a
noindexdirective on a page that crawlers are allowed to fetch. - It is per hostname. Your
www, non-www, and subdomains each need their own file. A rule onexample.comdoes not apply toblog.example.com.
These limits matter for AI search because many site owners assume that blocking a bot also removes their content from every AI product. It does not. It only stops that specific crawler from fetching pages going forward.
The AI Crawlers You Need to Know in 2026
The biggest mistake I see in audits is treating each AI company as one bot. In reality, each company runs multiple crawlers with different purposes. Blocking the wrong one gives you the opposite result from what you intended.
| Company | User-agent | Main purpose | Affects AI search visibility? |
|---|---|---|---|
| OpenAI | GPTBot | Collects content that may be used to train generative models | Not directly |
| OpenAI | OAI-SearchBot | Indexes pages for ChatGPT search results | Yes |
| OpenAI | ChatGPT-User | Fetches pages when a user asks ChatGPT to browse or act | Yes, for live answers |
| Anthropic | ClaudeBot | Collects content that may contribute to model training | Not directly |
| Anthropic | Claude-SearchBot | Improves search result quality for Claude users | Yes |
| Anthropic | Claude-User | Fetches pages when a user asks Claude to access a site | Yes, for live answers |
| Perplexity | PerplexityBot | Surfaces and links websites in Perplexity search results | Yes |
| Perplexity | Perplexity-User | Fetches pages for a user’s specific question | Yes, for live answers |
| Google-Extended | Product token controlling use of content for Gemini training and grounding | Gemini only, not Search |
AI companies update these lists regularly, so always verify names against the official sources: OpenAI’s crawler documentation, Anthropic’s crawler help article, and Perplexity’s bot documentation.
Training bots vs search bots vs user-triggered bots
It helps to sort every AI crawler into one of three buckets:
- Training crawlers (GPTBot, ClaudeBot) gather content that may be used to build or improve models. Blocking them does not usually remove you from live AI search answers.
- Search and index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes behind AI search products. These are the bots that decide whether your page can be cited.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because a person asked a question. Some providers state that robots.txt rules may not apply to these requests because the user, not an automated crawl, initiated them. Check each provider’s current documentation.
This split is the foundation of a smart configuration. You can often allow the search bots that earn you citations while limiting training bots, if that fits your content strategy.
Should You Allow or Block AI Crawlers?
There is no universal answer, so I use a simple decision framework with clients. Ask yourself three questions.
1. Is visibility in AI answers a business goal? If you sell products, services, software, or local offerings, being cited by ChatGPT, Perplexity and Claude is a growing discovery channel. In that case, allow the search bots. When I built the GEO strategy behind the MobiMatter case study, where Google AI Overview impressions grew 121% year over year, open and well-structured crawl access was a non-negotiable foundation.
2. Is your content your product? Publishers, paid research firms, course creators and data providers often have a stronger case for blocking training bots, since the content itself is the revenue source. A middle path is common: allow search bots for citations, block training bots, and gate premium content behind a login.
3. Do you have legal or licensing constraints? Some sites hold licensed images, user-generated content or regulated information (healthcare, legal, finance). Your legal team may require restrictions regardless of SEO upside.
Based on those answers, most sites land in one of four strategies:
- Open strategy: Allow all AI crawlers. Best for brands that want maximum AI visibility.
- Citation-first strategy: Allow search and user bots, block training bots. Best for publishers and content-driven brands.
- Selective strategy: Allow AI bots on public marketing pages, block them from private or premium sections.
- Closed strategy: Block all AI crawlers. Best for sensitive or licensed content, with the trade-off of reduced AI visibility.
Step-by-Step: How to Configure robots.txt for AI Bots
Step 1: Find and back up your current file
Open yourdomain.com/robots.txt in a browser and copy everything into a safe document. On WordPress, this file may be virtual (generated by your SEO plugin) or a physical file in your root directory. Check both so you do not create conflicting versions.
Step 2: Audit what is already there
Look for old rules that could conflict. I regularly find legacy blanket rules such as User-agent: * followed by Disallow: /, left over from a staging site, that silently block every crawler, AI or otherwise. Also look for plugin-added rules and wildcard patterns that match more than intended.
Step 3: Understand how groups work
Each crawler follows the single most specific User-agent group that matches it, and does not combine rules from other groups. This is the most misunderstood part of robots.txt. If you create a group for GPTBot, it will ignore your User-agent: * group entirely. That means any rules you want GPTBot to follow, such as blocking /wp-admin/, must be repeated inside its own group.
Step 4: Add one group per bot
Write each User-agent line followed by its Allow and Disallow rules, separated by a blank line from the next group. Keep paths relative to the root and case-sensitive.
Step 5: Add your sitemap
Add a Sitemap: line with the full URL of your XML sitemap. This helps every crawler, including AI bots, discover your best pages.
Step 6: Upload, test and monitor
Upload the file to your root, validate it (more on testing below), then watch your server logs for several weeks to confirm each bot behaves as expected.
Ready-to-Use robots.txt Templates
Copy the template that matches your strategy and adjust the paths for your site. Replace the sitemap URL with your own.
Template 1: Open strategy (allow all AI crawlers)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
Sitemap: https://yourdomain.com/sitemap_index.xmlThis is explicit, which makes your intent clear to anyone reading the file later. Technically, crawlers are allowed by default, but explicit groups make audits and future changes far easier.
Template 2: Citation-first strategy (allow search, block training)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Training crawlers: blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search and citation crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
Sitemap: https://yourdomain.com/sitemap_index.xmlThis is the setup I recommend most often for content-driven brands. It keeps you eligible for citations in ChatGPT search, Claude and Perplexity while opting out of general model training. One caution: blocking Google-Extended limits use of your content for Gemini, so weigh that against your goals. It does not affect your Google Search rankings.
Template 3: Selective strategy (protect premium sections)
User-agent: GPTBot
Disallow: /members/
Disallow: /courses/
Disallow: /research-reports/
Allow: /
User-agent: ClaudeBot
Disallow: /members/
Disallow: /courses/
Disallow: /research-reports/
Allow: /
User-agent: PerplexityBot
Disallow: /members/
Allow: /This lets AI crawlers read your public blog and product pages while keeping paid or private areas off limits. Remember that the real protection for premium content is a login wall. Robots.txt is only the polite first layer.
Template 4: Closed strategy (block all major AI crawlers)
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /Use this only when you accept the trade-off: your brand will be much less likely to be cited or recommended in those AI products.
How to Do This Without Hurting Your SEO
This is the part that worries most site owners, and rightly so. The good news is that AI bot rules are separate from your Google and Bing rules, as long as you avoid these mistakes.
Mistake 1: Using a wildcard group by accident
A rule like User-agent: * with Disallow: / blocks Googlebot too. Never use a global disallow on a live site. If you must block everything for a specific bot, name that bot explicitly.
Mistake 2: Editing the Googlebot or Bingbot groups
Your AI changes should be additive. Do not rewrite, merge or delete the Googlebot, Bingbot or * groups when you add AI-specific groups. Make AI edits below the existing rules and leave everything else untouched.
Mistake 3: Blocking CSS, JavaScript or image folders
Some old guides recommend disallowing /wp-includes/ or theme directories. Google needs to render your pages to understand them, and Google’s guidance on crawling and rendering is clear that blocking resources can harm how your page is evaluated. AI crawlers also benefit from clean access to your content.
Mistake 4: Confusing Google-Extended with Googlebot
Google-Extended is a standalone product token. Per Google’s crawler overview, it does not change your inclusion or ranking in Google Search. But it also does not control AI Overviews. Those are tied to Search itself. If you want to limit how snippets appear, you need snippet controls such as nosnippet, max-snippet or data-nosnippet, which can reduce visibility in classic results too.
Mistake 5: Relying on Crawl-delay
Some sites add Crawl-delay lines to slow AI bots. Google ignores this directive, and support elsewhere is inconsistent. If an AI bot is overloading your server, handle it with rate limiting at the CDN or firewall level.
Mistake 6: Blocking by accident through your CDN or firewall
This one is sneaky. Your robots.txt can say Allow, while Cloudflare, Sucuri, or another security layer blocks the same bot with a 403 error. If you are using bot protection, check the settings in your dashboard. Cloudflare’s bot management documentation explains how verified bots and AI crawler controls work. A crawler that gets blocked at the firewall never reads your robots.txt allowances at all.
Mistake 7: Forgetting to serve robots.txt correctly
Your file must return a 200 status code, be served as plain text, and be reachable without redirects to a login page. If it returns a 5xx error, Google may treat your site as fully disallowed for a period of time. Keep the file under 500 KiB, since content beyond Google’s limit is ignored.
Testing and Validating Your robots.txt
Never publish changes blind. Use this checklist every time.
- Check the live file. Visit
yourdomain.com/robots.txtand confirm the new rules appear, with no cached or duplicate version. - Run Google’s robots.txt report. In Google Search Console, the robots.txt report shows the version Google fetched, when it was fetched, and any parsing errors or warnings. Google caches robots.txt for up to about a day, so allow time before expecting changes to register.
- Test Googlebot access to key URLs. Use the URL Inspection tool on your homepage, top service pages and top blog posts. They should all say crawling is allowed.
- Test AI bot access with a header request. From a terminal you can simulate a bot by sending its user-agent string and checking that the response is a 200 and not a 403.
curl -I -A "GPTBot" https://yourdomain.com/
curl -I -A "ClaudeBot" https://yourdomain.com/
curl -I -A "PerplexityBot" https://yourdomain.com/- Inspect your server logs. After a few days, filter your access logs for each user-agent. You should see GPTBot, ClaudeBot and PerplexityBot requests with 200 status codes on pages you allowed, and no requests (or only requests for robots.txt) on pages you blocked.
- Spot-check AI products. Ask ChatGPT, Perplexity and Claude questions your brand should be able to answer, then see whether your site appears as a cited source.
A simple user-agent test is not perfect, since some bots publish IP ranges for verification and spoofed user-agents exist. For security-sensitive setups, verify bots through the published IP lists from each provider instead of trusting the user-agent string alone.
Beyond robots.txt: Making Your Site Truly AI-Ready
A clean robots.txt gets AI crawlers through the door. It does not make them cite you. Once access is correct, move on to these layers:
- Server-side rendering. Many AI crawlers do not execute JavaScript the way Googlebot does. If your key content loads only through client-side scripts, it may be invisible to them. Make sure the important text is in the initial HTML.
- Structured data. Schema markup such as Organization, Article, FAQPage, Product and LocalBusiness helps machines understand who you are and what each page covers.
- An llms.txt file. This emerging proposal, described at llmstxt.org, gives language models a curated, markdown-based map of your most important content. It is not a replacement for robots.txt, and adoption by AI providers is still uneven, so treat it as a low-cost supporting signal.
- Entity clarity and topical depth. AI models cite sources that are clear, factual and consistent about who they are. Strong About pages, author bios, consistent brand naming and well-organized topic clusters all help.
- Fast, stable hosting. Slow responses and frequent timeouts reduce crawl frequency from every bot.
You can find more tutorials on each of these topics on the aiseobyabhi blog, and the full list of what I cover for clients is on my services page.
Monitoring: Treat robots.txt as a Living File
AI crawler names and behaviors change quickly. New user-agents appear, existing ones are renamed, and providers add new documentation. I recommend a simple routine:
- Monthly: Review server logs for new or unknown AI user-agents hitting your site.
- Quarterly: Re-read the official documentation from OpenAI, Anthropic and Perplexity, and compare it to your file.
- After every site migration, CMS update or security plugin change: Re-test your robots.txt and your firewall rules. Migrations are the most common cause of accidental site-wide blocks.
- Version control: Keep dated copies of every robots.txt version, with a short note on why each change was made. When traffic or citations change, this history saves hours of guesswork.
If you also want to see whether the changes are working, track AI Overview appearances, referral traffic from AI platforms in your analytics, and citation mentions across ChatGPT, Perplexity and Claude. I use this kind of reporting across client accounts, including the programs shown in my case studies.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT?
Not necessarily. GPTBot is OpenAI’s training-related crawler. OAI-SearchBot is the crawler tied to ChatGPT search results, so a site can block GPTBot and still be eligible for ChatGPT citations. Confirm the current behavior in OpenAI’s documentation, since policies evolve.
Will blocking AI bots hurt my Google rankings?
No. AI bot rules like GPTBot, ClaudeBot and PerplexityBot are separate from Googlebot. As long as you do not modify your Googlebot or wildcard groups, your Google rankings should not be affected by AI bot rules.
Is PerplexityBot used for model training?
Perplexity states that PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for AI foundation models. Read their bot documentation for the latest wording.
Do AI crawlers always obey robots.txt?
Major providers say their crawlers respect it, but compliance is voluntary and user-triggered fetchers may be treated differently. If you need hard enforcement, combine robots.txt with firewall rules, bot management and authentication.
Should I add AI bot rules to every subdomain?
Yes. Each subdomain has its own robots.txt file, so replicate your strategy where it applies, such as blog, shop or help center subdomains.
How long until my robots.txt changes take effect?
Google typically refreshes its cached copy within about a day. AI crawlers vary, so expect days to weeks before you see the change reflected in logs and citations.
Is robots.txt enough to protect premium content from AI training?
No. It is a polite request. Premium content should sit behind authentication, and contracts or licensing terms should cover legal protection.
Final Thoughts
Configuring robots.txt for AI is not about choosing between open and closed. It is about deciding which crawlers earn value for your business and giving each one the right instructions. In most cases, that means keeping your Googlebot and Bingbot rules untouched, allowing the search and user-facing AI bots that drive citations, and making a deliberate choice about training crawlers.
Get the file right, test it thoroughly, and pair it with solid technical foundations such as rendering, schema and entity clarity. That is how you earn citations in AI answers without giving up the rankings you already have.
Not sure whether your current robots.txt is helping or hurting your AI visibility? Book a free 20-minute consultation and I’ll review your setup and show you exactly where you may be losing citations. You can also learn more about me and my approach or browse more guides on the blog.