Technical · 6 min read

GPTBot, ClaudeBot, PerplexityBot: which AI crawlers should you allow?

A practical guide to AI crawlers, what each one does, and how to configure robots.txt and your firewall without losing AI visibility.

By the AEO GEO Marketing team · Updated

Key takeaways
  • AI companies run different crawlers for search, user-requested fetching and model training.
  • Blocking search and user crawlers removes you from AI answers; blocking training crawlers does not.
  • Many sites block AI crawlers accidentally through CDN or firewall settings.
  • robots.txt is a request, not enforcement; firewalls do the enforcing.
  • Review your choices yearly as new crawlers appear.

Why AI crawlers matter

AI engines can only cite pages they can access. Each major AI company operates crawlers that fetch web pages, and your robots.txt file and security settings decide whether they are allowed in. Get this wrong and you can disappear from ChatGPT, Perplexity or Claude answers without realising why.

At the same time, some businesses have legitimate reasons to limit how their content is used for training AI models. The good news is that most AI companies now separate crawlers by purpose, so you can make different choices for search and training.

The three kinds of AI crawler

Search and answer crawlers

These index pages so they can appear as sources in AI search results. Examples include OpenAI’s OAI-SearchBot, Anthropic’s Claude-SearchBot and PerplexityBot. Blocking them reduces or removes your visibility in those products.

User-triggered fetchers

These fetch a page when a user asks an assistant to read a specific URL or when the assistant browses in real time. Examples include ChatGPT-User, Claude-User and Perplexity-User. Blocking them stops assistants reading your pages on request.

Training crawlers

These collect content that may be used to train AI models. Examples include GPTBot, ClaudeBot, CCBot (Common Crawl, used by many AI developers), Applebot-Extended and Google-Extended (a control token rather than a separate crawler). Blocking them affects future model training but not current search citations from the same company.

Major crawlers at a glance

  • Googlebot: Google Search, including AI Overviews and AI Mode. Never block if you want search traffic.
  • Google-Extended: controls use of content for Gemini model training and grounding in some products; does not affect Google Search.
  • Bingbot: Bing Search, which also feeds Microsoft Copilot and is used by other AI search products.
  • OAI-SearchBot / ChatGPT-User / GPTBot: OpenAI search, user fetching and training.
  • Claude-SearchBot / Claude-User / ClaudeBot: Anthropic search, user fetching and training.
  • PerplexityBot / Perplexity-User: Perplexity indexing and user fetching.
  • Applebot / Applebot-Extended: Apple search features and AI training control.
  • Meta-ExternalAgent: Meta AI crawling.
  • Amazonbot: Amazon services including Alexa answers.
  • CCBot: Common Crawl’s open dataset, widely used for AI research and training.

Crawler names and purposes change, so check each company’s documentation before making final decisions.

Choosing a policy

Most businesses that want to be visible in AI answers choose one of two policies:

Maximum visibility

Allow all search, user and training crawlers. Your brand is understood by future models and citable in current AI search. This suits most service businesses, agencies, SaaS companies and local businesses whose goal is discovery.

Visible but not trained on

Allow search and user crawlers, block training crawlers. You stay citable in AI search while limiting use of your content for training. This suits publishers, research providers and businesses whose content is their product.

Blocking everything is rarely wise unless your content is behind a paywall or you have specific legal reasons.

How to write the robots.txt rules

robots.txt groups rules by user agent. A visibility-first file lists the AI crawlers explicitly with Allow rules, followed by a general rule for everyone else and a link to your sitemap. To block training while allowing search, add separate groups for training crawlers with a Disallow rule for the whole site, while keeping Allow rules for search and user agents.

Keep the file simple, test it with a robots.txt validator and remember that rules are matched by the most specific user agent group. Our own robots.txt lists each AI crawler by name so the intention is unambiguous.

Check your firewall and CDN

robots.txt is a polite request. Your CDN or firewall is what actually lets bots in or keeps them out. Several popular services offer one-click options to block AI bots, and some enable them by default for new sites. These settings can override everything in robots.txt.

Review bot management settings in services such as Cloudflare, Akamai, Fastly or your hosting provider. Look for categories like “AI crawlers” or “AI scrapers”, and make sure the bots you want are allowed. Check server logs or analytics for visits from these user agents to confirm access.

How to verify crawlers are genuine

Some scrapers pretend to be well-known bots. Major AI companies publish IP ranges or verification methods for their crawlers. If you see heavy traffic claiming to be a known bot, verify it against the published ranges before allowing or blocking. Rate limiting is a better tool than blanket blocking for managing load.

Common mistakes

  • Blocking OAI-SearchBot or PerplexityBot while wanting AI visibility.
  • Assuming Google-Extended affects Google Search rankings.
  • Forgetting that firewall settings override robots.txt.
  • Using wildcard rules that unintentionally block important directories.
  • Never revisiting the policy as new crawlers appear.

A quick audit

  1. Open yoursite.com/robots.txt and list every user agent rule.
  2. Compare against the crawlers above and decide your policy.
  3. Check your CDN and firewall bot settings.
  4. Search server logs for AI user agents and confirm successful responses.
  5. Test key pages with each engine by asking it about a specific URL.
  6. Document your policy and review it every year.

What happens when you block the wrong crawler

Blocking a search or user crawler has immediate, visible effects. Your pages stop appearing as sources in that engine’s answers, and when a user asks the assistant to read one of your URLs, it may report that it cannot access the page. Over time, the engine may describe your business using third-party sources instead, which can be outdated or inaccurate.

Blocking a training crawler has slower, less visible effects. New versions of the model may know less about your brand, but current search citations from the same company are not affected.

Crawlers and server load

Some site owners block AI crawlers because of server load. Reputable crawlers generally respect crawl-delay signals and adapt to server responses, and load problems are usually better solved with caching, a CDN and rate limiting than with blanket blocks. If a specific bot is causing problems, contact the operator through their published channel or limit its request rate, rather than removing your site from AI search entirely.

Some publishers and content businesses license their content to AI companies or want to negotiate before allowing training use. If your content is your product, such as research, courses or journalism, take advice on licensing and choose a policy that protects that value. Most service businesses, by contrast, benefit from their marketing content being widely understood and cited, because it brings customers.

Example policies by business type

  • Local service business: allow all search, user and training crawlers for maximum visibility.
  • B2B SaaS: allow everything for marketing pages and documentation; protect logged-in app areas with authentication, not robots.txt.
  • Publisher or research firm: allow search and user crawlers; block training crawlers unless licensed.
  • E-commerce: allow search and user crawlers for product and category pages; block internal search results and cart pages for all bots to save crawl budget.

Protecting private areas properly

robots.txt is public and only asks crawlers to stay away. It is not a security tool, and listing private folders in it can even advertise them. Customer portals, admin panels, staging sites and confidential documents should be protected with authentication, and sensitive pages should use noindex where they must remain publicly reachable. Keep robots.txt for managing crawl access to public content, and use proper access controls for anything that must stay private.

Keeping your policy up to date

New AI crawlers appear regularly and existing ones are renamed or split by purpose. Put a reminder in your calendar to review your robots.txt and firewall settings at least once a year, and whenever you notice a new AI product gaining traction in your market. Keep a short internal note explaining your policy so future team members understand why each rule exists.

Key terms in this guide

  • AI Overviews: AI-generated summaries that appear at the top of some Google search results, with links to the web pages used as sources.
  • AI Mode: A conversational search experience in Google that answers complex questions with AI-generated responses and follow-up questions, citing web sources.
  • Citation: A link or reference to a source shown inside an AI-generated answer. Citations are the main unit of success in GEO.
  • Grounding: Basing an AI-generated answer on retrieved sources rather than only on the model’s training data, which reduces errors and produces citations.
  • robots.txt: A file that tells crawlers which parts of a website they may access. It can allow or block specific AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.
  • GPTBot / OAI-SearchBot: OpenAI’s crawlers. GPTBot collects content that may be used for model training; OAI-SearchBot indexes pages for ChatGPT search results.
  • ClaudeBot: Anthropic’s web crawler, used to gather publicly available content for its Claude models and related search features.
  • PerplexityBot: Perplexity’s crawler, which indexes pages so they can appear as sources in Perplexity answers.

How AEO GEO Marketing can help

If you would rather have specialists handle this, these services cover the work described in this guide:

AI SEO & Generative Engine Optimization

Get cited in Google AI Overviews, ChatGPT & Perplexity. The new search visibility standard. It includes geo / aeo full audit, ai overview citation strategy, entity authority building and more, from $800/mo.

Web Design & Development

Performance-first websites built for conversion, speed and AI search visibility. It includes ux & visual design, core web vitals engineering, schema-ready architecture and more, from $1,500/mo.

AI Visibility Audit

A full report on how AI engines see your brand, with a prioritised 90-day action plan. It includes prompt testing, competitor benchmark, technical ai readiness and more, from $750 one-off.

Not sure where to start? The free 48-hour AI snapshot shows where AI engines cite you today.

Frequently asked questions

If I block GPTBot, will ChatGPT stop citing me?

Not necessarily. ChatGPT search uses OAI-SearchBot. Blocking GPTBot affects training, not search citations.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended does not affect Google Search, including AI Overviews.

Is robots.txt legally binding?

robots.txt is a voluntary standard. Reputable AI companies state that they honour it; enforcement against bad actors requires technical controls.

Should I allow CCBot?

Common Crawl data is used widely in AI research and model training. Allow it if broad visibility matters more to you than limiting training use.

Continue with these guides:

Or see how we put this into practice:

Want this done for you?

Get a free 48-hour GEO audit that shows exactly where AI engines cite you — and where they cite your competitors.