Mare Liberum · open data · the state of the agent web

Which top websites block Google-Extended?

27% of the 289 most visited websites read so far block Google-Extended (Google) in robots.txt — 77 sites. Another 32 turn all automated readers away.

Google-Extended collects pages to train AI models. Blocking it keeps the site out of future training data; it does not stop assistants from reading the site for a person.

Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)

Sites that block Google-Extended

repubblica.it85/100 Ablocks 7 of 18 agents
alibaba.com80/100 Bblocks 2 of 18 agents
straitstimes.com80/100 Bblocks 1 of 18 agents
chatgpt.com78/100 Bblocks 4 of 18 agents
espn.com78/100 Bblocks 5 of 18 agents
abc.net.au75/100 Bblocks 8 of 18 agents
idnes.cz75/100 Bblocks 6 of 18 agents
tagesschau.de75/100 Bblocks 7 of 18 agents
ansa.it71/100 Bblocks 10 of 18 agents
lefigaro.fr71/100 Bblocks 7 of 18 agents
globo.com70/100 Bblocks 3 of 18 agents
yahoo.com70/100 Bblocks 7 of 18 agents
yle.fi70/100 Bblocks 8 of 18 agents
smh.com.au69/100 Cblocks 12 of 18 agents
svt.se68/100 Cblocks 8 of 18 agents
nzz.ch66/100 Cblocks 13 of 18 agents
sueddeutsche.de65/100 Cblocks 5 of 18 agents
whatsapp.com65/100 Cblocks 6 of 18 agents
asahi.com64/100 Cblocks 14 of 18 agents
bbc.co.uk64/100 Cblocks 14 of 18 agents
bbc.com64/100 Cblocks 14 of 18 agents
hln.be64/100 Cblocks 14 of 18 agents
theverge.com64/100 Cblocks 13 of 18 agents
uol.com.br64/100 Cblocks 13 of 18 agents
aftonbladet.se63/100 Cblocks 9 of 18 agents
derstandard.at63/100 Cblocks 10 of 18 agents
emol.com63/100 Cblocks 7 of 18 agents
nos.nl63/100 Cblocks 10 of 18 agents
orf.at63/100 Cblocks 8 of 18 agents
vg.no63/100 Cblocks 9 of 18 agents
corriere.it62/100 Cblocks 15 of 18 agents
dailymail.co.uk60/100 Cblocks 16 of 18 agents
dr.dk60/100 Cblocks 16 of 18 agents
elmundo.es60/100 Cblocks 16 of 18 agents
francetvinfo.fr60/100 Cblocks 16 of 18 agents
gazzetta.it60/100 Cblocks 16 of 18 agents
marca.com60/100 Cblocks 16 of 18 agents
nu.nl60/100 Cblocks 16 of 18 agents
publico.pt60/100 Cblocks 16 of 18 agents
rappler.com60/100 Cblocks 16 of 18 agents
yahoo.co.jp60/100 Cblocks 2 of 18 agents
lemonde.fr59/100 Cblocks 10 of 18 agents
kathimerini.gr58/100 Cblocks 10 of 18 agents
faz.net57/100 Cblocks 15 of 18 agents
nrk.no52/100 Cblocks 15 of 18 agents
pinterest.com52/100 Cblocks 15 of 18 agents
nzherald.co.nz50/100 Cblocks 16 of 18 agents
theglobeandmail.com50/100 Cblocks 16 of 18 agents
cnbc.com44/100 Dblocks 14 of 18 agents
wired.com44/100 Dblocks 13 of 18 agents
amazon.com42/100 Dblocks 14 of 18 agents
tiktok.com42/100 Dblocks 15 of 18 agents
x.com40/100 Dblocks 16 of 18 agents
baidu.com37/100 Dblocks 17 of 18 agents
stuff.co.nz36/100 Dblocks 11 of 18 agents
facebook.com30/100 Dblocks 16 of 18 agents
instagram.com30/100 Dblocks 16 of 18 agents
aliexpress.com28/100 Eblocks 6 of 18 agents
tripadvisor.com28/100 Eblocks 8 of 18 agents
claude.ai25/100 Eblocks 4 of 18 agents
ft.com25/100 Eblocks 8 of 18 agents
ahram.org.eg23/100 Eblocks 8 of 18 agents
bloomberg.com23/100 Eblocks 10 of 18 agents
glassdoor.com23/100 Eblocks 7 of 18 agents
kicker.de21/100 Eblocks 11 of 18 agents
amazon.de17/100 Eblocks 14 of 18 agents
economist.com16/100 Eblocks 12 of 18 agents
nation.africa14/100 Eblocks 12 of 18 agents
nytimes.com14/100 Eblocks 14 of 18 agents
reuters.com14/100 Eblocks 14 of 18 agents
wsj.com14/100 Eblocks 13 of 18 agents
news24.com12/100 Eblocks 15 of 18 agents
news.com.au10/100 Eblocks 16 of 18 agents
zeit.de10/100 Eblocks 16 of 18 agents
imdb.com9/100 Eblocks 14 of 18 agents
linkedin.com9/100 Eblocks 14 of 18 agents
reddit.com0/100 Eblocks 18 of 18 agents

Is your site one of them?

Check any website — free, no account: mareliberum.dev/check

The other agents

Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1% · Googlebot 0%

robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto