Mare Liberum · open data · the state of the agent web
Which top websites block Claude-SearchBot?
18% of the 289 most visited websites read so far block Claude-SearchBot (Anthropic) in robots.txt — 53 sites. Another 32 turn all automated readers away.
Claude-SearchBot builds the index an AI search or answer engine draws from. Blocking it keeps the site out of those answers.
Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)
Sites that block Claude-SearchBot
| lefigaro.fr | 71/100 B | blocks 7 of 18 agents |
| spiegel.de | 69/100 C | blocks 11 of 18 agents |
| asahi.com | 64/100 C | blocks 14 of 18 agents |
| bbc.co.uk | 64/100 C | blocks 14 of 18 agents |
| bbc.com | 64/100 C | blocks 14 of 18 agents |
| theverge.com | 64/100 C | blocks 13 of 18 agents |
| uol.com.br | 64/100 C | blocks 13 of 18 agents |
| aftonbladet.se | 63/100 C | blocks 9 of 18 agents |
| vg.no | 63/100 C | blocks 9 of 18 agents |
| corriere.it | 62/100 C | blocks 15 of 18 agents |
| dailymail.co.uk | 60/100 C | blocks 16 of 18 agents |
| dr.dk | 60/100 C | blocks 16 of 18 agents |
| elmundo.es | 60/100 C | blocks 16 of 18 agents |
| eluniversal.com.mx | 60/100 C | blocks 15 of 18 agents |
| francetvinfo.fr | 60/100 C | blocks 16 of 18 agents |
| gazzetta.it | 60/100 C | blocks 16 of 18 agents |
| kompas.com | 60/100 C | blocks 15 of 18 agents |
| marca.com | 60/100 C | blocks 16 of 18 agents |
| nu.nl | 60/100 C | blocks 16 of 18 agents |
| publico.pt | 60/100 C | blocks 16 of 18 agents |
| rappler.com | 60/100 C | blocks 16 of 18 agents |
| telegraph.co.uk | 60/100 C | blocks 15 of 18 agents |
| lemonde.fr | 59/100 C | blocks 10 of 18 agents |
| faz.net | 57/100 C | blocks 15 of 18 agents |
| theguardian.com | 54/100 C | blocks 10 of 18 agents |
| nrk.no | 52/100 C | blocks 15 of 18 agents |
| pinterest.com | 52/100 C | blocks 15 of 18 agents |
| cnn.com | 50/100 C | blocks 14 of 18 agents |
| nzherald.co.nz | 50/100 C | blocks 16 of 18 agents |
| theglobeandmail.com | 50/100 C | blocks 16 of 18 agents |
| cnbc.com | 44/100 D | blocks 14 of 18 agents |
| wired.com | 44/100 D | blocks 13 of 18 agents |
| amazon.com | 42/100 D | blocks 14 of 18 agents |
| tiktok.com | 42/100 D | blocks 15 of 18 agents |
| x.com | 40/100 D | blocks 16 of 18 agents |
| baidu.com | 37/100 D | blocks 17 of 18 agents |
| facebook.com | 30/100 D | blocks 16 of 18 agents |
| instagram.com | 30/100 D | blocks 16 of 18 agents |
| kicker.de | 21/100 E | blocks 11 of 18 agents |
| apnews.com | 19/100 E | blocks 9 of 18 agents |
| amazon.de | 17/100 E | blocks 14 of 18 agents |
| economist.com | 16/100 E | blocks 12 of 18 agents |
| nytimes.com | 14/100 E | blocks 14 of 18 agents |
| quora.com | 14/100 E | blocks 9 of 18 agents |
| realestate.com.au | 14/100 E | blocks 11 of 18 agents |
| reuters.com | 14/100 E | blocks 14 of 18 agents |
| wsj.com | 14/100 E | blocks 13 of 18 agents |
| news24.com | 12/100 E | blocks 15 of 18 agents |
| news.com.au | 10/100 E | blocks 16 of 18 agents |
| zeit.de | 10/100 E | blocks 16 of 18 agents |
| elpais.com | 9/100 E | blocks 10 of 18 agents |
| imdb.com | 9/100 E | blocks 14 of 18 agents |
| reddit.com | 0/100 E | blocks 18 of 18 agents |
Is your site one of them?
Check any website — free, no account: mareliberum.dev/check
The other agents
Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-User 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1% · Googlebot 0%
robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto