# Which AI crawlers matter, and which just take your content

> The major AI companies now run separate bots for training, for search, and for answering a user in the moment. Blocking the wrong one can make you invisible. Here is which is which, and the Google trap most sites fall into.

**Côme Chevalier**, Product Manager, CrawlerPal · September 22, 2026 · Technical GEO

Canonical: https://crawlerpal.com/blog/which-ai-crawlers-matter

---

More bot traffic is not more visibility. The AI companies now run several crawlers each, doing different jobs, and which one you block decides whether you are protecting your content or quietly removing yourself from AI answers.

## Three jobs, not one bot

Across the major vendors the pattern is the same. Each runs separate bots for separate purposes, and each can be controlled separately in robots.txt.

| Job | What it does | What blocking it costs you |
| --- | --- | --- |
| Training | Collects content to train future models | Your content stops feeding future models. Today's answers are unaffected |
| Search indexing | Builds the index an engine searches when it answers | You drop out of that engine's search answers |
| User fetch | Visits a page because a person just asked a question | The engine cannot read your page for that answer |

## Who runs what

These are the names each vendor documents.

| Vendor | Training | Search indexing | User fetch |
| --- | --- | --- | --- |
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None documented | PerplexityBot | Perplexity-User |

One detail matters for Perplexity. Its documentation says Perplexity-User generally ignores robots.txt, because each fetch is tied to a person's request. PerplexityBot, the indexing crawler, does respect it.

## The Google trap

Google works differently, and this is where a lot of sites get it wrong.

Google-Extended is not a crawler. It is a token you use in robots.txt to say whether Google may use your content to train its Gemini models. It never appears in your logs, because the crawling is still done by Google's ordinary user agents.

Blocking Google-Extended does not remove you from AI Overviews. Those are built on Google's normal search index. Keeping pages out of them has traditionally meant restricting snippets, which also affects your ordinary search results, so check Google's current documentation before you change anything, because these controls have been moving.

So a site that blocks Google-Extended thinking it has left AI search is still in AI Overviews. All it has done is stop contributing to Gemini training.

## What I would do

Decide on training separately from visibility. Blocking the training bots is a legitimate choice about your content, and it costs you nothing in today's answers.

Leave the search and user-fetch bots open if you want to be found. Blocking them is how a site quietly removes itself from the engines its buyers use.

Then read your robots.txt against the table above. A rule written years ago for a different reason, or a blanket block on anything that looks like a bot, is a common way to be invisible without meaning to be.

## Frequently asked questions

### Should I block AI crawlers?

It depends which ones. Blocking training bots such as GPTBot or ClaudeBot stops your content feeding future models without affecting today's answers. Blocking search and user-fetch bots such as OAI-SearchBot or Claude-User removes you from those engines' answers.

### Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended is a robots.txt token that controls whether your content is used to train Gemini models. AI Overviews are built on Google's normal search index, so blocking Google-Extended leaves you in them.

### Why does my site get lots of AI bot traffic but no citations?

Much of it is likely training crawlers, which collect content for future models rather than answering questions today. Heavy crawl activity and being cited in answers are different things, so a traffic spike on its own says little about visibility.

### Do all AI crawlers respect robots.txt?

Most documented crawlers do. One notable exception is Perplexity-User, which Perplexity says generally ignores robots.txt because each fetch is tied to a person's request. Its indexing crawler, PerplexityBot, does respect it.

---

CrawlerPal · https://crawlerpal.com