AI crawlers: who really comes — counted live
On 2026-11-21, AI crawlers made 105,501 of the 123,056 requests to this address. These are the names behind them, as they introduce themselves:
| Who | Kind | Requests |
|---|---|---|
| Meta | ai-crawler | 66,527 |
| Anthropic ClaudeBot | ai-crawler | 29,909 |
| ai-crawler | 4,761 | |
| OpenAI GPTBot | ai-crawler | 4,304 |
| Yandex | search | 425 |
| search | 411 | |
| Claude | assistant | 111 |
| Bing | search | 84 |
| OpenAI search | search-ai | 7 |
| Perplexity | search-ai | 1 |
Every number, every day: the census · /census (JSON)
The AI crawlers and their robots.txt names
| robots.txt | Operator | What for |
|---|---|---|
GPTBot | OpenAI | collects pages for training OpenAI models |
OAI-SearchBot | OpenAI | indexes pages for ChatGPT search |
ChatGPT-User | OpenAI | fetches a page when a ChatGPT user asks for it |
ClaudeBot | Anthropic | collects pages for training Claude |
Claude-SearchBot | Anthropic | indexes pages for Claude’s web search |
Claude-User | Anthropic | fetches a page when a Claude user asks for it |
PerplexityBot | Perplexity | indexes pages for Perplexity answers |
Perplexity-User | Perplexity | fetches a page for a Perplexity user |
Google-Extended | a robots.txt token only: decides whether Gemini may train on pages Googlebot fetched | |
meta-externalagent | Meta | collects pages for Meta’s AI models |
Applebot-Extended | Apple | a robots.txt token: decides whether Apple may train on pages Applebot fetched |
Amazonbot | Amazon | crawls for Amazon services including Alexa answers |
Bytespider | ByteDance | collects pages for ByteDance models |
CCBot | Common Crawl | builds the open Common Crawl corpus that many models train on |
DuckAssistBot | DuckDuckGo | fetches pages for DuckDuckGo’s AI answers |
MistralAI-User | Mistral | fetches a page when a Le Chat user asks for it |
robots.txt: appear in answers, keep out of training
# allow AI search and user fetches User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # keep out of training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot User-agent: Bytespider Disallow: /
Whether your website is open to these crawlers, the free check shows: AI visibility check.
Questions
Which AI crawlers are there?
The main ones: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, meta-externalagent (Meta), Amazonbot, Bytespider, CCBot, and the robots.txt tokens Google-Extended and Applebot-Extended.
Should I block AI crawlers?
To appear in answers from ChatGPT, Claude or Perplexity, let the search and user crawlers in. Training can be blocked separately, for example GPTBot, ClaudeBot and Google-Extended.
Where do these numbers come from?
From the public census of Mare Liberum: every request to this address, sorted by who it says it is, with no record of people. Day: 2026-11-21.
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto
Free MCP server · MCP server directory · AI visibility check · Web page to Markdown · AI crawlers, live · For companies · Deutsch