AI Crawler Bot User-Agent List: Who Visits Your Site and Why

AI crawlers are bots that visit websites for AI companies. Some collect pages for model training, some build search indexes for AI answers, and some fetch a page live when a user asks a question. Knowing their user-agent names lets you choose in robots.txt which ones may access your site. Blocking all of them can reduce your chances of being found and cited in AI answers.

The Three Types of AI Bots

• Training crawlers collect public pages that may be used to train future models.
• Search crawlers build an index that an AI product uses to find and cite sources.
• User-triggered fetchers visit a page because a person asked the assistant to look something up right now.

Many companies now use separate user-agents for each purpose, so you can, for example, allow search but not training.

Common AI User-Agents

Names change, so always confirm against each company’s own documentation before you edit your rules. At the time of writing, commonly documented names include:

OpenAI

• GPTBot: crawling for model training.
• OAI-SearchBot: crawling for search features in ChatGPT.
• ChatGPT-User: pages fetched when a user’s request needs them.

Anthropic

• ClaudeBot: crawling for model training.
• Claude-SearchBot and Claude-User: search indexing and user-requested fetching.

Google

• Googlebot: the main Google Search crawler, which also feeds AI Overviews.
• Google-Extended: a robots.txt token, not a separate bot, that controls whether content may be used for Gemini model training and grounding. Blocking it does not remove you from Google Search.

Microsoft

• Bingbot: the Bing crawler. Copilot relies on Bing, so blocking Bingbot can limit your presence there.

Perplexity

• PerplexityBot: indexing for Perplexity’s answers.
• Perplexity-User: fetches made on behalf of a user.

Others

• Applebot and Applebot-Extended (Apple: search features, and a token for AI training use).
• CCBot (Common Crawl, a public dataset widely used in AI research).
• Meta-ExternalAgent (Meta), Amazonbot (Amazon), Bytespider (ByteDance).

Example Robots.txt Rules

To allow AI search bots but block one training crawler, you might write:

User-agent: GPTBot then Disallow: /

User-agent: OAI-SearchBot then Allow: /

To keep a private area closed to every bot, add Disallow: /account/ under User-agent: *. Remember that robots.txt is a request, not a lock. Reputable crawlers respect it; it is not a security tool.

How to Decide What to Allow

1. List your goals. If you want to be recommended in AI answers, search and user-triggered bots usually need access to your public pages.
2. Protect what is private. Block login areas, carts and internal search results for everyone.
3. Decide on training separately. This is a business and rights decision. Allowing training crawlers may help models know your brand over time; blocking them is a valid choice too.
4. Check your firewall and CDN. Some security settings block AI bots by default, even if robots.txt allows them.

For the wider technical picture, read technical SEO for AI crawlers.

Check Your Server Logs

Your hosting logs show which bots actually visit. Look for the names above, how often they come and which pages they request. Be aware that some bad bots fake popular user-agents. Major providers publish IP ranges or verification methods you can use to confirm genuine visits.

How Keboost Helps

Access is only the first step. Keboost tracks whether AI assistants actually mention and cite your brand for the prompts your customers ask, and helps you create content worth citing. Run a free AI visibility check to see your starting point.