Mare Liberum · open data · the state of the agent web

Which top websites block ChatGPT-User?

22% of the 289 most visited websites read so far block ChatGPT-User (OpenAI) in robots.txt — 63 sites. Another 32 turn all automated readers away.

ChatGPT-User fetches a page right now because a person asked an assistant about it. Blocking it means the assistant cannot read the site for that person.

Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)

Sites that block ChatGPT-User

espn.com78/100 Bblocks 5 of 18 agents
abc.net.au75/100 Bblocks 8 of 18 agents
aljazeera.com75/100 Bblocks 5 of 18 agents
ansa.it71/100 Bblocks 10 of 18 agents
yahoo.com70/100 Bblocks 7 of 18 agents
yle.fi70/100 Bblocks 8 of 18 agents
smh.com.au69/100 Cblocks 12 of 18 agents
spiegel.de69/100 Cblocks 11 of 18 agents
hurriyet.com.tr68/100 Cblocks 5 of 18 agents
nzz.ch66/100 Cblocks 13 of 18 agents
sueddeutsche.de65/100 Cblocks 5 of 18 agents
asahi.com64/100 Cblocks 14 of 18 agents
bbc.co.uk64/100 Cblocks 14 of 18 agents
bbc.com64/100 Cblocks 14 of 18 agents
hln.be64/100 Cblocks 14 of 18 agents
mercadolibre.com.mx64/100 Cblocks 7 of 18 agents
mercadolivre.com.br64/100 Cblocks 7 of 18 agents
theverge.com64/100 Cblocks 13 of 18 agents
derstandard.at63/100 Cblocks 10 of 18 agents
eltiempo.com63/100 Cblocks 8 of 18 agents
emol.com63/100 Cblocks 7 of 18 agents
nos.nl63/100 Cblocks 10 of 18 agents
corriere.it62/100 Cblocks 15 of 18 agents
dailymail.co.uk60/100 Cblocks 16 of 18 agents
dr.dk60/100 Cblocks 16 of 18 agents
elmundo.es60/100 Cblocks 16 of 18 agents
eluniversal.com.mx60/100 Cblocks 15 of 18 agents
francetvinfo.fr60/100 Cblocks 16 of 18 agents
gazzetta.it60/100 Cblocks 16 of 18 agents
kompas.com60/100 Cblocks 15 of 18 agents
marca.com60/100 Cblocks 16 of 18 agents
nu.nl60/100 Cblocks 16 of 18 agents
publico.pt60/100 Cblocks 16 of 18 agents
rappler.com60/100 Cblocks 16 of 18 agents
telegraph.co.uk60/100 Cblocks 15 of 18 agents
kathimerini.gr58/100 Cblocks 10 of 18 agents
faz.net57/100 Cblocks 15 of 18 agents
nrk.no52/100 Cblocks 15 of 18 agents
pinterest.com52/100 Cblocks 15 of 18 agents
cnn.com50/100 Cblocks 14 of 18 agents
nzherald.co.nz50/100 Cblocks 16 of 18 agents
theglobeandmail.com50/100 Cblocks 16 of 18 agents
cnbc.com44/100 Dblocks 14 of 18 agents
amazon.com42/100 Dblocks 14 of 18 agents
tiktok.com42/100 Dblocks 15 of 18 agents
x.com40/100 Dblocks 16 of 18 agents
baidu.com37/100 Dblocks 17 of 18 agents
stuff.co.nz36/100 Dblocks 11 of 18 agents
facebook.com30/100 Dblocks 16 of 18 agents
instagram.com30/100 Dblocks 16 of 18 agents
claude.ai25/100 Eblocks 4 of 18 agents
kicker.de21/100 Eblocks 11 of 18 agents
amazon.de17/100 Eblocks 14 of 18 agents
economist.com16/100 Eblocks 12 of 18 agents
nation.africa14/100 Eblocks 12 of 18 agents
nytimes.com14/100 Eblocks 14 of 18 agents
quora.com14/100 Eblocks 9 of 18 agents
news24.com12/100 Eblocks 15 of 18 agents
news.com.au10/100 Eblocks 16 of 18 agents
zeit.de10/100 Eblocks 16 of 18 agents
imdb.com9/100 Eblocks 14 of 18 agents
linkedin.com9/100 Eblocks 14 of 18 agents
reddit.com0/100 Eblocks 18 of 18 agents

Is your site one of them?

Check any website — free, no account: mareliberum.dev/check

The other agents

Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1% · Googlebot 0%

robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto