Mare Liberum · open data · the state of the agent web

Which top websites block Claude-User?

18% of the 289 most visited websites read so far block Claude-User (Anthropic) in robots.txt — 53 sites. Another 32 turn all automated readers away.

Claude-User fetches a page right now because a person asked an assistant about it. Blocking it means the assistant cannot read the site for that person.

Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)

Sites that block Claude-User

lefigaro.fr71/100 Bblocks 7 of 18 agents
spiegel.de69/100 Cblocks 11 of 18 agents
asahi.com64/100 Cblocks 14 of 18 agents
hln.be64/100 Cblocks 14 of 18 agents
mercadolibre.com.mx64/100 Cblocks 7 of 18 agents
mercadolivre.com.br64/100 Cblocks 7 of 18 agents
theverge.com64/100 Cblocks 13 of 18 agents
uol.com.br64/100 Cblocks 13 of 18 agents
corriere.it62/100 Cblocks 15 of 18 agents
dailymail.co.uk60/100 Cblocks 16 of 18 agents
dr.dk60/100 Cblocks 16 of 18 agents
elmundo.es60/100 Cblocks 16 of 18 agents
eluniversal.com.mx60/100 Cblocks 15 of 18 agents
francetvinfo.fr60/100 Cblocks 16 of 18 agents
gazzetta.it60/100 Cblocks 16 of 18 agents
kompas.com60/100 Cblocks 15 of 18 agents
marca.com60/100 Cblocks 16 of 18 agents
nu.nl60/100 Cblocks 16 of 18 agents
publico.pt60/100 Cblocks 16 of 18 agents
rappler.com60/100 Cblocks 16 of 18 agents
telegraph.co.uk60/100 Cblocks 15 of 18 agents
lemonde.fr59/100 Cblocks 10 of 18 agents
faz.net57/100 Cblocks 15 of 18 agents
theguardian.com54/100 Cblocks 10 of 18 agents
nrk.no52/100 Cblocks 15 of 18 agents
pinterest.com52/100 Cblocks 15 of 18 agents
cnn.com50/100 Cblocks 14 of 18 agents
nzherald.co.nz50/100 Cblocks 16 of 18 agents
theglobeandmail.com50/100 Cblocks 16 of 18 agents
cnbc.com44/100 Dblocks 14 of 18 agents
wired.com44/100 Dblocks 13 of 18 agents
amazon.com42/100 Dblocks 14 of 18 agents
tiktok.com42/100 Dblocks 15 of 18 agents
x.com40/100 Dblocks 16 of 18 agents
baidu.com37/100 Dblocks 17 of 18 agents
facebook.com30/100 Dblocks 16 of 18 agents
instagram.com30/100 Dblocks 16 of 18 agents
apnews.com19/100 Eblocks 9 of 18 agents
amazon.de17/100 Eblocks 14 of 18 agents
economist.com16/100 Eblocks 12 of 18 agents
nation.africa14/100 Eblocks 12 of 18 agents
nytimes.com14/100 Eblocks 14 of 18 agents
quora.com14/100 Eblocks 9 of 18 agents
realestate.com.au14/100 Eblocks 11 of 18 agents
reuters.com14/100 Eblocks 14 of 18 agents
wsj.com14/100 Eblocks 13 of 18 agents
news24.com12/100 Eblocks 15 of 18 agents
news.com.au10/100 Eblocks 16 of 18 agents
zeit.de10/100 Eblocks 16 of 18 agents
elpais.com9/100 Eblocks 10 of 18 agents
imdb.com9/100 Eblocks 14 of 18 agents
linkedin.com9/100 Eblocks 14 of 18 agents
reddit.com0/100 Eblocks 18 of 18 agents

Is your site one of them?

Check any website — free, no account: mareliberum.dev/check

The other agents

Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1% · Googlebot 0%

robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto