Deutsch

AI crawlers: who really comes — counted live

On 2026-11-21, AI crawlers made 105,501 of the 123,056 requests to this address. These are the names behind them, as they introduce themselves:

WhoKindRequests
Metaai-crawler66,527
Anthropic ClaudeBotai-crawler29,909
Googleai-crawler4,761
OpenAI GPTBotai-crawler4,304
Yandexsearch425
Googlesearch411
Claudeassistant111
Bingsearch84
OpenAI searchsearch-ai7
Perplexitysearch-ai1

Every number, every day: the census · /census (JSON)

The AI crawlers and their robots.txt names

robots.txtOperatorWhat for
GPTBotOpenAIcollects pages for training OpenAI models
OAI-SearchBotOpenAIindexes pages for ChatGPT search
ChatGPT-UserOpenAIfetches a page when a ChatGPT user asks for it
ClaudeBotAnthropiccollects pages for training Claude
Claude-SearchBotAnthropicindexes pages for Claude’s web search
Claude-UserAnthropicfetches a page when a Claude user asks for it
PerplexityBotPerplexityindexes pages for Perplexity answers
Perplexity-UserPerplexityfetches a page for a Perplexity user
Google-ExtendedGooglea robots.txt token only: decides whether Gemini may train on pages Googlebot fetched
meta-externalagentMetacollects pages for Meta’s AI models
Applebot-ExtendedApplea robots.txt token: decides whether Apple may train on pages Applebot fetched
AmazonbotAmazoncrawls for Amazon services including Alexa answers
BytespiderByteDancecollects pages for ByteDance models
CCBotCommon Crawlbuilds the open Common Crawl corpus that many models train on
DuckAssistBotDuckDuckGofetches pages for DuckDuckGo’s AI answers
MistralAI-UserMistralfetches a page when a Le Chat user asks for it

robots.txt: appear in answers, keep out of training

# allow AI search and user fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# keep out of training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Whether your website is open to these crawlers, the free check shows: AI visibility check.

Questions

Which AI crawlers are there?

The main ones: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, meta-externalagent (Meta), Amazonbot, Bytespider, CCBot, and the robots.txt tokens Google-Extended and Applebot-Extended.

Should I block AI crawlers?

To appear in answers from ChatGPT, Claude or Perplexity, let the search and user crawlers in. Training can be blocked separately, for example GPTBot, ClaudeBot and Google-Extended.

Where do these numbers come from?

From the public census of Mare Liberum: every request to this address, sorted by who it says it is, with no record of people. Day: 2026-11-21.

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto

Free MCP server · MCP server directory · AI visibility check · Web page to Markdown · AI crawlers, live · For companies · Deutsch