Should you block AI crawlers?

• Training crawlers collect pages that may be used to train future models. Blocking them is a statement about how your content is used. It doesn’t change whether you can be found.
• Search crawlers build the index an assistant searches when it answers with citations. Block one and that assistant has nothing of yours to cite.
• User-triggered fetchers load a single page because a person asked the assistant about it or pasted its URL. They act more like a browser than a crawler, and two of the companies say robots.txt may not apply to them.

Two tokens, Google-Extended and Applebot-Extended, are not crawlers at all. They are labels in robots.txt that control how pages gathered by the company’s ordinary crawler may be used.

OpenAI

• GPTBot: crawls content that may be used to train OpenAI’s generative AI foundation models. Disallowing it signals the site’s content should not be used for that.
• OAI-SearchBot: surfaces websites in ChatGPT’s search features. Sites that opt out won’t be shown in ChatGPT search answers, though they can still appear as navigational links.
• ChatGPT-User: visits a page when a user’s action in ChatGPT or a custom GPT requires it. It isn’t used for automatic crawling, and OpenAI says robots.txt rules may not apply because a user initiated the request.

OpenAI states that each setting is independent, so a site can allow OAI-SearchBot and disallow GPTBot (OpenAI’s crawler docs).

Anthropic

• ClaudeBot: collects web content that could contribute to model training. Blocking it signals that the site’s future material should be excluded from training datasets.
• Claude-SearchBot: crawls to improve search results for Claude users. Blocking it stops Anthropic indexing your content for search and may reduce your visibility there.
• Claude-User: fetches pages when a Claude user’s question calls for it. Blocking it prevents retrieval of your content in response to a user query.

Anthropic says its bots honour robots.txt and the non-standard Crawl-delay directive, and warns that blocking by IP address may not work as an opt-out because it stops the bots reading your robots.txt (Anthropic’s crawler article).

Perplexity

• PerplexityBot: surfaces and links websites in Perplexity’s search results. Perplexity says it is not used to crawl content for AI foundation models.
• Perplexity-User: fetches a page when a user’s question needs it. Perplexity says this fetcher generally ignores robots.txt because a user requested the fetch.

Source: Perplexity’s crawler docs.

Google

• Googlebot: the Search crawler. Google says robots.txt rules for Googlebot are the control for AI Overviews and AI Mode, because those features are part of Search (source). There is no separate token that removes you from AI Overviews while keeping you in ordinary results. Snippet controls such as nosnippet and max-snippet limit what is shown.
• Google-Extended: a control token with no user-agent string of its own. It governs whether content Google has crawled may be used to train future Gemini models, and for grounding in Gemini Apps and in Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal (Google’s crawler list).

Apple

• Applebot: powers search features in Spotlight, Siri and Safari.
• Applebot-Extended: doesn’t crawl. Disallowing it opts your content out of training Apple’s generative models, and pages that disallow it can still be included in search results (Apple’s Applebot page).

Common Crawl

• CCBot: the crawler of a nonprofit that publishes an open repository of web crawl data (Common Crawl’s CCBot page). Because the dataset is open, once a page is in it you have no say over who downloads it or what they do with it.

Bing and Microsoft

• Bingbot: the Bing search crawler. Microsoft’s AI controls are usage preferences addressed to Bingbot, not a second crawler token. On 30 October 2026 it announced two: AI Training, set in robots.txt with the token __usage_ai-training or per page with a Bingbot meta tag, and AI Grounding, set per page only. Both default to allow, neither changes eligibility for ordinary Bing results, and the older noarchive and nocache tags stop being read as AI preferences after 1 March 2027 (Bing’s announcement).

The same __usage_ai-training token is meant to be shared. Google’s crawler page says it intends to support it in early 2027, and Anthropic says it honours a training preference signal based on the same IETF draft.

A publisher whose articles are the product has a reason to keep them out of training sets. A product site is in a different position: its pages are marketing, written to be read as widely as possible.

• Search crawlers: allow them. The documented effect of blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot is that you drop out of that assistant’s search answers. If you want to be found, there is nothing to gain.
• User-triggered fetchers: allow them. These requests come from someone who asked about your page. Blocking them mostly means the assistant can’t read a link your own prospect pasted in.
• Training crawlers: your call. The documented effect of blocking is only that your content is excluded from future training. Nothing in the docs says it hurts search inclusion. The undocumented side is what a model can say about you without searching: that can only come from training data, so a product nobody was allowed to train on is, by inference, less likely to be known that way. Nobody outside these companies can measure how much.
• Google-Extended is the awkward one. It covers Gemini training and also grounding in the Gemini apps, so disallowing it may limit how Gemini uses your pages when answering. It has no effect on Google Search or AI Overviews.

For most new products the sensible default is to allow everything, then block specific training tokens only if you have content worth protecting, such as paid courses or original research.

If you decide to refuse training and keep everything else, this is the whole file:

# Training crawlers and training opt-out tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: __usage_ai-training
Disallow: /

# Everyone else, including search crawlers and user fetchers
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://yoursite.com/sitemap.xml

Things to know before copying it:

• Remove the Google-Extended line if you want Gemini to be able to ground answers on your pages.
• Under the robots.txt standard, a crawler follows the group that names it and falls back to * only when no group does (RFC 9309). So if you write a group for a specific bot, repeat any path rules you want it to follow. A named group does not inherit Disallow: /admin/ from the * group.
• Don’t add OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot or Bingbot to the blocked group. A “block all AI bots” list copied from elsewhere may include them.
• OpenAI and Perplexity both say a change can take about 24 hours to be reflected. robots.txt applies per host, so a subdomain needs its own file.

The sitemap and robots.txt guide covers the rest of the file.

The standard says it in so many words: the rules are not a form of access authorization, and the protocol is not a substitute for valid content security measures (RFC 9309). The companies listed above say they honour it for their crawlers. That leaves three gaps:

• Fetchers that say it may not apply. ChatGPT-User and Perplexity-User, by their own documentation.
• Crawlers that don’t identify themselves. A bot with a browser user agent never matches your rules. Common Crawl warns that other crawlers falsely identify as CCBot, which shows the user-agent string alone proves little.
• Content already collected. Anthropic’s wording is that blocking ClaudeBot excludes future material. A rule added today doesn’t reach back.

If something must not be read, put it behind a login. If you need to enforce a block, do it at the firewall or CDN, and verify real crawlers against the IP ranges that OpenAI, Anthropic, Perplexity and Common Crawl publish. Keep robots.txt itself reachable, or compliant crawlers can’t read your rules.

1. Open yoursite.com/robots.txt and read every group. Look for a blanket Disallow: / left from staging, and for a pasted AI block list that includes search crawlers.
2. Check your host, CDN and firewall settings. A bot-blocking rule there works at the network level, where robots.txt has no say, so find out which user agents it covers.
3. Run a key page through the indexability checker to confirm ordinary search engines aren’t blocked.
4. Look in your server logs to see who actually visits. How to test whether AI assistants know your product has the command.

This area keeps moving: Bing’s new preferences were announced less than two weeks before this guide was written. Before you rely on any list, including this one, check it against each company’s own page.