Mare Liberum · open data · the state of the agent web
Which top websites block ChatGPT-User?
22% of the 289 most visited websites read so far block ChatGPT-User (OpenAI) in robots.txt — 63 sites. Another 32 turn all automated readers away.
ChatGPT-User fetches a page right now because a person asked an assistant about it. Blocking it means the assistant cannot read the site for that person.
Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)
Sites that block ChatGPT-User
| espn.com | 78/100 B | blocks 5 of 18 agents |
| abc.net.au | 75/100 B | blocks 8 of 18 agents |
| aljazeera.com | 75/100 B | blocks 5 of 18 agents |
| ansa.it | 71/100 B | blocks 10 of 18 agents |
| yahoo.com | 70/100 B | blocks 7 of 18 agents |
| yle.fi | 70/100 B | blocks 8 of 18 agents |
| smh.com.au | 69/100 C | blocks 12 of 18 agents |
| spiegel.de | 69/100 C | blocks 11 of 18 agents |
| hurriyet.com.tr | 68/100 C | blocks 5 of 18 agents |
| nzz.ch | 66/100 C | blocks 13 of 18 agents |
| sueddeutsche.de | 65/100 C | blocks 5 of 18 agents |
| asahi.com | 64/100 C | blocks 14 of 18 agents |
| bbc.co.uk | 64/100 C | blocks 14 of 18 agents |
| bbc.com | 64/100 C | blocks 14 of 18 agents |
| hln.be | 64/100 C | blocks 14 of 18 agents |
| mercadolibre.com.mx | 64/100 C | blocks 7 of 18 agents |
| mercadolivre.com.br | 64/100 C | blocks 7 of 18 agents |
| theverge.com | 64/100 C | blocks 13 of 18 agents |
| derstandard.at | 63/100 C | blocks 10 of 18 agents |
| eltiempo.com | 63/100 C | blocks 8 of 18 agents |
| emol.com | 63/100 C | blocks 7 of 18 agents |
| nos.nl | 63/100 C | blocks 10 of 18 agents |
| corriere.it | 62/100 C | blocks 15 of 18 agents |
| dailymail.co.uk | 60/100 C | blocks 16 of 18 agents |
| dr.dk | 60/100 C | blocks 16 of 18 agents |
| elmundo.es | 60/100 C | blocks 16 of 18 agents |
| eluniversal.com.mx | 60/100 C | blocks 15 of 18 agents |
| francetvinfo.fr | 60/100 C | blocks 16 of 18 agents |
| gazzetta.it | 60/100 C | blocks 16 of 18 agents |
| kompas.com | 60/100 C | blocks 15 of 18 agents |
| marca.com | 60/100 C | blocks 16 of 18 agents |
| nu.nl | 60/100 C | blocks 16 of 18 agents |
| publico.pt | 60/100 C | blocks 16 of 18 agents |
| rappler.com | 60/100 C | blocks 16 of 18 agents |
| telegraph.co.uk | 60/100 C | blocks 15 of 18 agents |
| kathimerini.gr | 58/100 C | blocks 10 of 18 agents |
| faz.net | 57/100 C | blocks 15 of 18 agents |
| nrk.no | 52/100 C | blocks 15 of 18 agents |
| pinterest.com | 52/100 C | blocks 15 of 18 agents |
| cnn.com | 50/100 C | blocks 14 of 18 agents |
| nzherald.co.nz | 50/100 C | blocks 16 of 18 agents |
| theglobeandmail.com | 50/100 C | blocks 16 of 18 agents |
| cnbc.com | 44/100 D | blocks 14 of 18 agents |
| amazon.com | 42/100 D | blocks 14 of 18 agents |
| tiktok.com | 42/100 D | blocks 15 of 18 agents |
| x.com | 40/100 D | blocks 16 of 18 agents |
| baidu.com | 37/100 D | blocks 17 of 18 agents |
| stuff.co.nz | 36/100 D | blocks 11 of 18 agents |
| facebook.com | 30/100 D | blocks 16 of 18 agents |
| instagram.com | 30/100 D | blocks 16 of 18 agents |
| claude.ai | 25/100 E | blocks 4 of 18 agents |
| kicker.de | 21/100 E | blocks 11 of 18 agents |
| amazon.de | 17/100 E | blocks 14 of 18 agents |
| economist.com | 16/100 E | blocks 12 of 18 agents |
| nation.africa | 14/100 E | blocks 12 of 18 agents |
| nytimes.com | 14/100 E | blocks 14 of 18 agents |
| quora.com | 14/100 E | blocks 9 of 18 agents |
| news24.com | 12/100 E | blocks 15 of 18 agents |
| news.com.au | 10/100 E | blocks 16 of 18 agents |
| zeit.de | 10/100 E | blocks 16 of 18 agents |
| imdb.com | 9/100 E | blocks 14 of 18 agents |
| linkedin.com | 9/100 E | blocks 14 of 18 agents |
| reddit.com | 0/100 E | blocks 18 of 18 agents |
Is your site one of them?
Check any website — free, no account: mareliberum.dev/check
The other agents
Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1% · Googlebot 0%
robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto