Mare Liberum · open data

The state of the agent web

Of the 289 most visited websites read so far, 43% block at least one AI agent in robots.txt, 28% block an assistant that reads for a person right now, and 11% turn automated readers away entirely. Only 16% publish an llms.txt and 0% an agent card.

Average score 52/100 · 289 of 303 sites read · updated 2026-11-19 22:51 UTC · /web.json (CC0)

Who gets shut out

AgentRoleBlocked byShare
Bytespider ByteDancetrains models100 sites
35%
ClaudeBot Anthropictrains models96 sites
33%
CCBot Common Crawltrains models96 sites
33%
GPTBot OpenAItrains models94 sites
33%
Applebot-Extended Appletrains models83 sites
29%
Meta-ExternalAgent Metatrains models82 sites
28%
PerplexityBot PerplexityAI search80 sites
28%
Amazonbot AmazonAI search78 sites
27%
Google-Extended Googletrains models77 sites
27%
ChatGPT-User OpenAIreads for a person63 sites
22%
Perplexity-User Perplexityreads for a person54 sites
19%
Claude-User Anthropicreads for a person53 sites
18%
Claude-SearchBot AnthropicAI search53 sites
18%
DuckAssistBot DuckDuckGoreads for a person50 sites
17%
OAI-SearchBot OpenAIAI search46 sites
16%
MistralAI-User Mistralreads for a person45 sites
16%
Bingbot MicrosoftAI search2 sites
1%
Googlebot GoogleAI search1 site
0%

Grades

A 22 · B 80 · C 78 · D 46 · E 63

By country domain

DE · 28 sites · ⌀ 52GB · 9 sites · ⌀ 52FR · 6 sites · ⌀ 55IT · 6 sites · ⌀ 54AU · 5 sites · ⌀ 39NL · 4 sites · ⌀ 73PL · 4 sites · ⌀ 62CH · 3 sites · ⌀ 71SE · 3 sites · ⌀ 70BR · 3 sites · ⌀ 69AT · 3 sites · ⌀ 65NO · 3 sites · ⌀ 65ES · 3 sites · ⌀ 47RU · 3 sites · ⌀ 33MX · 2 sites · ⌀ 62CA · 2 sites · ⌀ 53SG · 2 sites · ⌀ 45NZ · 2 sites · ⌀ 43

Every site

cloudflare.com100 Alets all in · llms.txt · agent card
adobe.com90 Alets all in · llms.txt
cohere.com90 Alets all in · llms.txt
foxnews.com90 Alets all in · llms.txt
hindustantimes.com90 Ablocks 1: Meta-ExternalAgent · llms.txt
hubspot.com90 Alets all in · llms.txt
intel.com90 Alets all in · llms.txt
marktplaats.nl90 Alets all in · llms.txt
mistral.ai90 Alets all in · llms.txt
nvidia.com90 Alets all in · llms.txt
paytm.com90 Alets all in · llms.txt
samsung.com90 Alets all in · llms.txt
shein.com90 Alets all in · llms.txt
squarespace.com90 Alets all in · llms.txt
stripe.com90 Alets all in · llms.txt
target.com90 Alets all in · llms.txt
time.com90 Alets all in · llms.txt
tum.de90 Alets all in · llms.txt
vercel.com90 Alets all in · llms.txt
notion.so88 Ablocks 1: Amazonbot · llms.txt
weather.com88 Ablocks 3: Amazonbot, CCBot, Bytespider · llms.txt
repubblica.it85 Ablocks 7: GPTBot, PerplexityBot, Google-Extended, Applebot-Extended … · llms.txt
airbnb.com80 Bblocks 3: GPTBot, ClaudeBot, Applebot-Extended
alibaba.com80 Bblocks 2: GPTBot, Google-Extended
apple.com80 Blets all in
atlassian.com80 Blets all in · llms.txt
blocket.se80 Blets all in
cam.ac.uk80 Blets all in
coursera.org80 Bblocks 1: Meta-ExternalAgent · llms.txt
detik.com80 Blets all in
digi24.ro80 Blets all in
discord.com80 Blets all in
dropbox.com80 Blets all in · llms.txt
finn.no80 Blets all in
github.com80 Bblocks 1: Bytespider · llms.txt
gov.br80 Blets all in
grab.com80 Blets all in
gulfnews.com80 Blets all in
harvard.edu80 Blets all in
heise.de80 Blets all in
indiatimes.com80 Bblocks 1: Meta-ExternalAgent
johnlewis.com80 Blets all in
lidl.de80 Blets all in
microsoft.com80 Blets all in
nasa.gov80 Blets all in
nike.com80 Blets all in
punchng.com80 Blets all in
rijksoverheid.nl80 Blets all in
rtve.es80 Blets all in
shopify.com80 Blets all in · llms.txt
sky.com80 Blets all in
slack.com80 Blets all in · llms.txt
sparkasse.de80 Blets all in
stability.ai80 Blets all in
straitstimes.com80 Bblocks 1: Google-Extended
techcrunch.com80 Bblocks 5: GPTBot, ClaudeBot, Applebot-Extended, CCBot …
temu.com80 Bblocks 1: Bytespider
thairath.co.th80 Blets all in
usa.gov80 Blets all in
walmart.com80 Blets all in
who.int80 Bblocks 1: CCBot
zdf.de80 Blets all in
zoom.us80 Blets all in
chatgpt.com78 Bblocks 4: PerplexityBot, Google-Extended, CCBot, Bytespider
espn.com78 Bblocks 5: ChatGPT-User, GPTBot, Google-Extended, CCBot …
ikea.com78 Bblocks 1: Amazonbot
srf.ch78 Bblocks 7: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent …
abc.net.au75 Bblocks 8: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
aljazeera.com75 Bblocks 5: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
idnes.cz75 Bblocks 6: GPTBot, ClaudeBot, PerplexityBot, Google-Extended …
kijiji.ca75 Blets all in · llms.txt
otto.de75 Blets all in · llms.txt
stepstone.de75 Blets all in
tagesschau.de75 Bblocks 7: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended …
wordpress.com75 Blets all in · llms.txt
onet.pl73 Bblocks 3: Meta-ExternalAgent, Amazonbot, Bytespider
t-online.de73 Bblocks 7: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended …
ansa.it71 Bblocks 10: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
lefigaro.fr71 Bblocks 7: Claude-User, Claude-SearchBot, ClaudeBot, Google-Extended …
admin.ch70 Blets all in
anthropic.com70 Blets all in
check24.de70 Blets all in
chosun.com70 Bblocks 1: GPTBot
clarin.com70 Blets all in
daum.net70 Blets all in
deepmind.google70 Blets all in · llms.txt
globo.com70 Bblocks 3: GPTBot, Google-Extended, CCBot
huggingface.co70 Blets all in
lanacion.com.ar70 Blets all in
nhs.uk70 Blets all in
qq.com70 Blets all in
salesforce.com70 Blets all in · little text without JavaScript · llms.txt
service-public.fr70 Blets all in
stanford.edu70 Blets all in
uber.com70 Blets all in
webmd.com70 Bblocks 3: GPTBot, ClaudeBot, CCBot
wikipedia.org70 Blets all in
willhaben.at70 Blets all in
wp.pl70 Bblocks 2: GPTBot, CCBot
yahoo.com70 Bblocks 7: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
yle.fi70 Bblocks 8: ChatGPT-User, GPTBot, ClaudeBot, Google-Extended …
youtube.com70 Blets all in
bild.de69 Cblocks 8: Perplexity-User, PerplexityBot, Applebot-Extended, Amazonbot …
smh.com.au69 Cblocks 12: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot …
spiegel.de69 Cblocks 11: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
healthline.com68 Cblocks 7: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent …
hurriyet.com.tr68 Cblocks 5: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot …
sap.com68 Cblocks 4: Applebot-Extended, Meta-ExternalAgent, DuckAssistBot, Bytespider · llms.txt
svt.se68 Cblocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended …
nzz.ch66 Cblocks 13: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot …
bahn.de65 Clets all in
bilibili.com65 Clets all in · llms.txt
cdc.gov65 Clets all in
costco.com65 Clets all in
dm.de65 Clets all in · little text without JavaScript · llms.txt
gov.pl65 Clets all in
gov.uk65 Clets all in
mit.edu65 Clets all in
mozilla.org65 Clets all in
rightmove.co.uk65 Cblocks 2: GPTBot, CCBot
sina.com.cn65 Clets all in
substack.com65 Clets all in · llms.txt
sueddeutsche.de65 Cblocks 5: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
w3.org65 Clets all in
whatsapp.com65 Cblocks 6: GPTBot, ClaudeBot, PerplexityBot, Google-Extended …
asahi.com64 Cblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
bbc.co.uk64 Cblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-SearchBot …
bbc.com64 Cblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-SearchBot …
hln.be64 Cblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
mercadolibre.com.mx64 Cblocks 7: ChatGPT-User, GPTBot, Claude-User, ClaudeBot …
mercadolivre.com.br64 Cblocks 7: ChatGPT-User, GPTBot, Claude-User, ClaudeBot …
theverge.com64 Cblocks 13: ChatGPT-User, Claude-User, Claude-SearchBot, ClaudeBot …
uol.com.br64 Cblocks 13: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User …
aftonbladet.se63 Cblocks 9: Claude-SearchBot, ClaudeBot, PerplexityBot, Google-Extended …
derstandard.at63 Cblocks 10: ChatGPT-User, GPTBot, ClaudeBot, Perplexity-User …
eltiempo.com63 Cblocks 8: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot …
emol.com63 Cblocks 7: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
golem.de63 Cblocks 2: Amazonbot, Bytespider
kakaku.com63 Cblocks 2: Amazonbot, Bytespider
nos.nl63 Cblocks 10: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
orf.at63 Cblocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended …
vg.no63 Cblocks 9: Claude-SearchBot, ClaudeBot, PerplexityBot, Google-Extended …
corriere.it62 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
dailymail.co.uk60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … · llms.txt
deepseek.com60 Clets all in
dr.dk60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
duckduckgo.com60 Clets all in
elmundo.es60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
eluniversal.com.mx60 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
francetvinfo.fr60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
gazzetta.it60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
kompas.com60 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
marca.com60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
meta.com60 Clets all in · little text without JavaScript
nu.nl60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
publico.pt60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
rappler.com60 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
telegram.org60 Clets all in
telegraph.co.uk60 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
twitch.tv60 Clets all in · little text without JavaScript
web.de60 Clets all in
yahoo.co.jp60 Cblocks 2: GPTBot, Google-Extended
lemonde.fr59 Cblocks 10: Claude-User, Claude-SearchBot, ClaudeBot, Google-Extended …
kathimerini.gr58 Cblocks 10: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot …
faz.net57 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
netflix.com56 Cblocks 8: Perplexity-User, Applebot-Extended, Meta-ExternalAgent, Amazonbot …
gov.sg55 Cturns readers away
taobao.com55 Clets all in
theguardian.com54 Cblocks 10: Claude-User, Claude-SearchBot, ClaudeBot, PerplexityBot …
nrk.no52 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
pinterest.com52 Cblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
bing.com50 Clets all in · little text without JavaScript
cnn.com50 Cblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
duolingo.com50 Clets all in · little text without JavaScript
index.hu50 Cturns readers away
nzherald.co.nz50 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
spotify.com50 Clets all in · little text without JavaScript
theglobeandmail.com50 Cblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
un.org50 Clets all in
zalando.de50 Clets all in · little text without JavaScript · llms.txt
google.com45 Dlets all in · little text without JavaScript
naver.com45 Dlets all in · little text without JavaScript
nikkei.com45 Dturns readers away
rai.it45 Dlets all in · little text without JavaScript
wix.com45 Dlets all in · little text without JavaScript · llms.txt
cnbc.com44 Dblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
wired.com44 Dblocks 13: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User …
amazon.com42 Dblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
tiktok.com42 Dblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
allegro.pl40 Dlets all in · llms.txt
archive.org40 Dlets all in · little text without JavaScript
bund.de40 Dlets all in · little text without JavaScript
cdiscount.com40 Dlets all in · little text without JavaScript
europa.eu40 Dlets all in
jd.com40 Dlets all in · little text without JavaScript
orange.fr40 Dlets all in · little text without JavaScript
paypal.com40 Dlets all in · llms.txt
trendyol.com40 Dlets all in · llms.txt
x.com40 Dblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
baidu.com37 Dblocks 17: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
stuff.co.nz36 Dblocks 11: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot … · little text without JavaScript
etsy.com35 Dlets all in · llms.txt
expedia.com35 Dlets all in · llms.txt
indeed.com35 Dlets all in · llms.txt
mail.ru35 Dlets all in · little text without JavaScript
shopee.sg35 Dlets all in · little text without JavaScript
vk.com35 Dlets all in · little text without JavaScript
yandex.ru35 Dlets all in · little text without JavaScript
avito.ru30 Dlets all in
bol.com30 Dlets all in
canada.ca30 Dlets all in · little text without JavaScript
canva.com30 Dblocks 6: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent …
character.ai30 Dlets all in
facebook.com30 Dblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
homedepot.com30 Dlets all in
ibm.com30 Dlets all in
instagram.com30 Dblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … · little text without JavaScript
jumia.com.ng30 Dblocks 1: CCBot
leboncoin.fr30 Dblocks 1: Bytespider
openai.com30 Dlets all in
perplexity.ai30 Dlets all in
rewe.de30 Dlets all in
takealot.com30 Dlets all in
timesofisrael.com30 Dlets all in
wayfair.com30 Dlets all in
x.ai30 Dlets all in
aliexpress.com28 Eblocks 6: GPTBot, ClaudeBot, Google-Extended, Amazonbot …
medium.com28 Eblocks 6: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent …
tripadvisor.com28 Eblocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended …
zillow.com28 Eblocks 3: Applebot-Extended, Meta-ExternalAgent, Amazonbot
claude.ai25 Eblocks 4: ChatGPT-User, OAI-SearchBot, GPTBot, Google-Extended
ebay.com25 Eblocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended …
ebay.de25 Eblocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended …
flipkart.com25 Eturns readers away
forbes.com25 Eblocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended …
ft.com25 Eblocks 8: ClaudeBot, Perplexity-User, PerplexityBot, Google-Extended …
idealista.com25 Elets all in
irctc.co.in25 Elets all in
seek.com.au25 Elets all in
vnexpress.net25 Elets all in
ahram.org.eg23 Eblocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended …
bloomberg.com23 Eblocks 10: GPTBot, ClaudeBot, PerplexityBot, Google-Extended …
glassdoor.com23 Eblocks 7: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended …
kicker.de21 Eblocks 11: ChatGPT-User, GPTBot, Claude-SearchBot, ClaudeBot …
chefkoch.de20 Eblocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended …
apnews.com19 Eblocks 9: GPTBot, Claude-User, Claude-SearchBot, ClaudeBot …
amazon.de17 Eblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … · little text without JavaScript
economist.com16 Eblocks 12: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
khanacademy.org15 Eturns readers away · little text without JavaScript
nation.africa14 Eblocks 12: ChatGPT-User, GPTBot, Claude-User, ClaudeBot …
nytimes.com14 Eblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
quora.com14 Eblocks 9: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
realestate.com.au14 Eblocks 11: GPTBot, Claude-User, Claude-SearchBot, ClaudeBot …
reuters.com14 Eblocks 14: GPTBot, Claude-User, Claude-SearchBot, ClaudeBot …
wsj.com14 Eblocks 13: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User …
news24.com12 Eblocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
lowes.com10 Eturns readers away · little text without JavaScript
news.com.au10 Eblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
zeit.de10 Eblocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
elpais.com9 Eblocks 10: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User …
imdb.com9 Eblocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
linkedin.com9 Eblocks 14: ChatGPT-User, GPTBot, Claude-User, ClaudeBot …
adidas.com0 Eturns readers away
argos.co.uk0 Eturns readers away
booking.com0 Eturns readers away
coupang.com0 Eturns readers away
dawn.com0 Eturns readers away
elcorteingles.es0 Eturns readers away
fnac.com0 Eturns readers away
haaretz.co.il0 Eturns readers away
hm.com0 Eturns readers away
idealo.de0 Eturns readers away
macys.com0 Eturns readers away
mayoclinic.org0 Eturns readers away
mediamarkt.de0 Eturns readers away
mercedes-benz.de0 Eturns readers away
midjourney.com0 Eturns readers away
ndtv.com0 Eturns readers away
nih.gov0 Eturns readers away
oracle.com0 Eturns readers away
ox.ac.uk0 Eturns readers away
reddit.com0 Eblocks 18: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User …
rozetka.com.ua0 Eturns readers away
sony.com0 Eturns readers away
stackoverflow.com0 Eturns readers away
subito.it0 Eturns readers away
tesco.com0 Eturns readers away
zara.com0 Eturns readers away
zomato.com0 Eturns readers away

This data is free for everyone. Support the open atlas — voluntary, any amount.

Your site

Check any website — free, no account: mareliberum.dev/check

the most visited websites of the world, read again every six days from what each publishes (robots.txt, llms.txt, home page, agent card); open data, CC0. robots.txt states wishes; most large AI operators say they follow it. A site that turns automated readers away also stops most AI assistants.

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto