Mare Liberum · open data
The state of the agent web
Of the 289 most visited websites read so far, 43% block at least one AI agent in robots.txt, 28% block an assistant that reads for a person right now, and 11% turn automated readers away entirely. Only 16% publish an llms.txt and 0% an agent card.
Average score 52/100 · 289 of 303 sites read · updated 2026-11-19 22:51 UTC · /web.json (CC0)
Who gets shut out
| Agent | Role | Blocked by | Share | |
|---|---|---|---|---|
| Bytespider ByteDance | trains models | 100 sites | 35% | |
| ClaudeBot Anthropic | trains models | 96 sites | 33% | |
| CCBot Common Crawl | trains models | 96 sites | 33% | |
| GPTBot OpenAI | trains models | 94 sites | 33% | |
| Applebot-Extended Apple | trains models | 83 sites | 29% | |
| Meta-ExternalAgent Meta | trains models | 82 sites | 28% | |
| PerplexityBot Perplexity | AI search | 80 sites | 28% | |
| Amazonbot Amazon | AI search | 78 sites | 27% | |
| Google-Extended Google | trains models | 77 sites | 27% | |
| ChatGPT-User OpenAI | reads for a person | 63 sites | 22% | |
| Perplexity-User Perplexity | reads for a person | 54 sites | 19% | |
| Claude-User Anthropic | reads for a person | 53 sites | 18% | |
| Claude-SearchBot Anthropic | AI search | 53 sites | 18% | |
| DuckAssistBot DuckDuckGo | reads for a person | 50 sites | 17% | |
| OAI-SearchBot OpenAI | AI search | 46 sites | 16% | |
| MistralAI-User Mistral | reads for a person | 45 sites | 16% | |
| Bingbot Microsoft | AI search | 2 sites | 1% | |
| Googlebot Google | AI search | 1 site | 0% |
Grades
A 22 · B 80 · C 78 · D 46 · E 63
By country domain
DE · 28 sites · ⌀ 52GB · 9 sites · ⌀ 52FR · 6 sites · ⌀ 55IT · 6 sites · ⌀ 54AU · 5 sites · ⌀ 39NL · 4 sites · ⌀ 73PL · 4 sites · ⌀ 62CH · 3 sites · ⌀ 71SE · 3 sites · ⌀ 70BR · 3 sites · ⌀ 69AT · 3 sites · ⌀ 65NO · 3 sites · ⌀ 65ES · 3 sites · ⌀ 47RU · 3 sites · ⌀ 33MX · 2 sites · ⌀ 62CA · 2 sites · ⌀ 53SG · 2 sites · ⌀ 45NZ · 2 sites · ⌀ 43
Every site
| cloudflare.com | 100 A | lets all in · llms.txt · agent card |
| adobe.com | 90 A | lets all in · llms.txt |
| cohere.com | 90 A | lets all in · llms.txt |
| foxnews.com | 90 A | lets all in · llms.txt |
| hindustantimes.com | 90 A | blocks 1: Meta-ExternalAgent · llms.txt |
| hubspot.com | 90 A | lets all in · llms.txt |
| intel.com | 90 A | lets all in · llms.txt |
| marktplaats.nl | 90 A | lets all in · llms.txt |
| mistral.ai | 90 A | lets all in · llms.txt |
| nvidia.com | 90 A | lets all in · llms.txt |
| paytm.com | 90 A | lets all in · llms.txt |
| samsung.com | 90 A | lets all in · llms.txt |
| shein.com | 90 A | lets all in · llms.txt |
| squarespace.com | 90 A | lets all in · llms.txt |
| stripe.com | 90 A | lets all in · llms.txt |
| target.com | 90 A | lets all in · llms.txt |
| time.com | 90 A | lets all in · llms.txt |
| tum.de | 90 A | lets all in · llms.txt |
| vercel.com | 90 A | lets all in · llms.txt |
| notion.so | 88 A | blocks 1: Amazonbot · llms.txt |
| weather.com | 88 A | blocks 3: Amazonbot, CCBot, Bytespider · llms.txt |
| repubblica.it | 85 A | blocks 7: GPTBot, PerplexityBot, Google-Extended, Applebot-Extended … · llms.txt |
| airbnb.com | 80 B | blocks 3: GPTBot, ClaudeBot, Applebot-Extended |
| alibaba.com | 80 B | blocks 2: GPTBot, Google-Extended |
| apple.com | 80 B | lets all in |
| atlassian.com | 80 B | lets all in · llms.txt |
| blocket.se | 80 B | lets all in |
| cam.ac.uk | 80 B | lets all in |
| coursera.org | 80 B | blocks 1: Meta-ExternalAgent · llms.txt |
| detik.com | 80 B | lets all in |
| digi24.ro | 80 B | lets all in |
| discord.com | 80 B | lets all in |
| dropbox.com | 80 B | lets all in · llms.txt |
| finn.no | 80 B | lets all in |
| github.com | 80 B | blocks 1: Bytespider · llms.txt |
| gov.br | 80 B | lets all in |
| grab.com | 80 B | lets all in |
| gulfnews.com | 80 B | lets all in |
| harvard.edu | 80 B | lets all in |
| heise.de | 80 B | lets all in |
| indiatimes.com | 80 B | blocks 1: Meta-ExternalAgent |
| johnlewis.com | 80 B | lets all in |
| lidl.de | 80 B | lets all in |
| microsoft.com | 80 B | lets all in |
| nasa.gov | 80 B | lets all in |
| nike.com | 80 B | lets all in |
| punchng.com | 80 B | lets all in |
| rijksoverheid.nl | 80 B | lets all in |
| rtve.es | 80 B | lets all in |
| shopify.com | 80 B | lets all in · llms.txt |
| sky.com | 80 B | lets all in |
| slack.com | 80 B | lets all in · llms.txt |
| sparkasse.de | 80 B | lets all in |
| stability.ai | 80 B | lets all in |
| straitstimes.com | 80 B | blocks 1: Google-Extended |
| techcrunch.com | 80 B | blocks 5: GPTBot, ClaudeBot, Applebot-Extended, CCBot … |
| temu.com | 80 B | blocks 1: Bytespider |
| thairath.co.th | 80 B | lets all in |
| usa.gov | 80 B | lets all in |
| walmart.com | 80 B | lets all in |
| who.int | 80 B | blocks 1: CCBot |
| zdf.de | 80 B | lets all in |
| zoom.us | 80 B | lets all in |
| chatgpt.com | 78 B | blocks 4: PerplexityBot, Google-Extended, CCBot, Bytespider |
| espn.com | 78 B | blocks 5: ChatGPT-User, GPTBot, Google-Extended, CCBot … |
| ikea.com | 78 B | blocks 1: Amazonbot |
| srf.ch | 78 B | blocks 7: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent … |
| abc.net.au | 75 B | blocks 8: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| aljazeera.com | 75 B | blocks 5: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| idnes.cz | 75 B | blocks 6: GPTBot, ClaudeBot, PerplexityBot, Google-Extended … |
| kijiji.ca | 75 B | lets all in · llms.txt |
| otto.de | 75 B | lets all in · llms.txt |
| stepstone.de | 75 B | lets all in |
| tagesschau.de | 75 B | blocks 7: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended … |
| wordpress.com | 75 B | lets all in · llms.txt |
| onet.pl | 73 B | blocks 3: Meta-ExternalAgent, Amazonbot, Bytespider |
| t-online.de | 73 B | blocks 7: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended … |
| ansa.it | 71 B | blocks 10: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| lefigaro.fr | 71 B | blocks 7: Claude-User, Claude-SearchBot, ClaudeBot, Google-Extended … |
| admin.ch | 70 B | lets all in |
| anthropic.com | 70 B | lets all in |
| check24.de | 70 B | lets all in |
| chosun.com | 70 B | blocks 1: GPTBot |
| clarin.com | 70 B | lets all in |
| daum.net | 70 B | lets all in |
| deepmind.google | 70 B | lets all in · llms.txt |
| globo.com | 70 B | blocks 3: GPTBot, Google-Extended, CCBot |
| huggingface.co | 70 B | lets all in |
| lanacion.com.ar | 70 B | lets all in |
| nhs.uk | 70 B | lets all in |
| qq.com | 70 B | lets all in |
| salesforce.com | 70 B | lets all in · little text without JavaScript · llms.txt |
| service-public.fr | 70 B | lets all in |
| stanford.edu | 70 B | lets all in |
| uber.com | 70 B | lets all in |
| webmd.com | 70 B | blocks 3: GPTBot, ClaudeBot, CCBot |
| wikipedia.org | 70 B | lets all in |
| willhaben.at | 70 B | lets all in |
| wp.pl | 70 B | blocks 2: GPTBot, CCBot |
| yahoo.com | 70 B | blocks 7: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| yle.fi | 70 B | blocks 8: ChatGPT-User, GPTBot, ClaudeBot, Google-Extended … |
| youtube.com | 70 B | lets all in |
| bild.de | 69 C | blocks 8: Perplexity-User, PerplexityBot, Applebot-Extended, Amazonbot … |
| smh.com.au | 69 C | blocks 12: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot … |
| spiegel.de | 69 C | blocks 11: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| healthline.com | 68 C | blocks 7: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent … |
| hurriyet.com.tr | 68 C | blocks 5: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot … |
| sap.com | 68 C | blocks 4: Applebot-Extended, Meta-ExternalAgent, DuckAssistBot, Bytespider · llms.txt |
| svt.se | 68 C | blocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended … |
| nzz.ch | 66 C | blocks 13: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot … |
| bahn.de | 65 C | lets all in |
| bilibili.com | 65 C | lets all in · llms.txt |
| cdc.gov | 65 C | lets all in |
| costco.com | 65 C | lets all in |
| dm.de | 65 C | lets all in · little text without JavaScript · llms.txt |
| gov.pl | 65 C | lets all in |
| gov.uk | 65 C | lets all in |
| mit.edu | 65 C | lets all in |
| mozilla.org | 65 C | lets all in |
| rightmove.co.uk | 65 C | blocks 2: GPTBot, CCBot |
| sina.com.cn | 65 C | lets all in |
| substack.com | 65 C | lets all in · llms.txt |
| sueddeutsche.de | 65 C | blocks 5: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| w3.org | 65 C | lets all in |
| whatsapp.com | 65 C | blocks 6: GPTBot, ClaudeBot, PerplexityBot, Google-Extended … |
| asahi.com | 64 C | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| bbc.co.uk | 64 C | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-SearchBot … |
| bbc.com | 64 C | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-SearchBot … |
| hln.be | 64 C | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| mercadolibre.com.mx | 64 C | blocks 7: ChatGPT-User, GPTBot, Claude-User, ClaudeBot … |
| mercadolivre.com.br | 64 C | blocks 7: ChatGPT-User, GPTBot, Claude-User, ClaudeBot … |
| theverge.com | 64 C | blocks 13: ChatGPT-User, Claude-User, Claude-SearchBot, ClaudeBot … |
| uol.com.br | 64 C | blocks 13: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User … |
| aftonbladet.se | 63 C | blocks 9: Claude-SearchBot, ClaudeBot, PerplexityBot, Google-Extended … |
| derstandard.at | 63 C | blocks 10: ChatGPT-User, GPTBot, ClaudeBot, Perplexity-User … |
| eltiempo.com | 63 C | blocks 8: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot … |
| emol.com | 63 C | blocks 7: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| golem.de | 63 C | blocks 2: Amazonbot, Bytespider |
| kakaku.com | 63 C | blocks 2: Amazonbot, Bytespider |
| nos.nl | 63 C | blocks 10: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| orf.at | 63 C | blocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended … |
| vg.no | 63 C | blocks 9: Claude-SearchBot, ClaudeBot, PerplexityBot, Google-Extended … |
| corriere.it | 62 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| dailymail.co.uk | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … · llms.txt |
| deepseek.com | 60 C | lets all in |
| dr.dk | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| duckduckgo.com | 60 C | lets all in |
| elmundo.es | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| eluniversal.com.mx | 60 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| francetvinfo.fr | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| gazzetta.it | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| kompas.com | 60 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| marca.com | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| meta.com | 60 C | lets all in · little text without JavaScript |
| nu.nl | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| publico.pt | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| rappler.com | 60 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| telegram.org | 60 C | lets all in |
| telegraph.co.uk | 60 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| twitch.tv | 60 C | lets all in · little text without JavaScript |
| web.de | 60 C | lets all in |
| yahoo.co.jp | 60 C | blocks 2: GPTBot, Google-Extended |
| lemonde.fr | 59 C | blocks 10: Claude-User, Claude-SearchBot, ClaudeBot, Google-Extended … |
| kathimerini.gr | 58 C | blocks 10: ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot … |
| faz.net | 57 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| netflix.com | 56 C | blocks 8: Perplexity-User, Applebot-Extended, Meta-ExternalAgent, Amazonbot … |
| gov.sg | 55 C | turns readers away |
| taobao.com | 55 C | lets all in |
| theguardian.com | 54 C | blocks 10: Claude-User, Claude-SearchBot, ClaudeBot, PerplexityBot … |
| nrk.no | 52 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| pinterest.com | 52 C | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| bing.com | 50 C | lets all in · little text without JavaScript |
| cnn.com | 50 C | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| duolingo.com | 50 C | lets all in · little text without JavaScript |
| index.hu | 50 C | turns readers away |
| nzherald.co.nz | 50 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| spotify.com | 50 C | lets all in · little text without JavaScript |
| theglobeandmail.com | 50 C | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| un.org | 50 C | lets all in |
| zalando.de | 50 C | lets all in · little text without JavaScript · llms.txt |
| google.com | 45 D | lets all in · little text without JavaScript |
| naver.com | 45 D | lets all in · little text without JavaScript |
| nikkei.com | 45 D | turns readers away |
| rai.it | 45 D | lets all in · little text without JavaScript |
| wix.com | 45 D | lets all in · little text without JavaScript · llms.txt |
| cnbc.com | 44 D | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| wired.com | 44 D | blocks 13: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User … |
| amazon.com | 42 D | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| tiktok.com | 42 D | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| allegro.pl | 40 D | lets all in · llms.txt |
| archive.org | 40 D | lets all in · little text without JavaScript |
| bund.de | 40 D | lets all in · little text without JavaScript |
| cdiscount.com | 40 D | lets all in · little text without JavaScript |
| europa.eu | 40 D | lets all in |
| jd.com | 40 D | lets all in · little text without JavaScript |
| orange.fr | 40 D | lets all in · little text without JavaScript |
| paypal.com | 40 D | lets all in · llms.txt |
| trendyol.com | 40 D | lets all in · llms.txt |
| x.com | 40 D | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| baidu.com | 37 D | blocks 17: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| stuff.co.nz | 36 D | blocks 11: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot … · little text without JavaScript |
| etsy.com | 35 D | lets all in · llms.txt |
| expedia.com | 35 D | lets all in · llms.txt |
| indeed.com | 35 D | lets all in · llms.txt |
| mail.ru | 35 D | lets all in · little text without JavaScript |
| shopee.sg | 35 D | lets all in · little text without JavaScript |
| vk.com | 35 D | lets all in · little text without JavaScript |
| yandex.ru | 35 D | lets all in · little text without JavaScript |
| avito.ru | 30 D | lets all in |
| bol.com | 30 D | lets all in |
| canada.ca | 30 D | lets all in · little text without JavaScript |
| canva.com | 30 D | blocks 6: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent … |
| character.ai | 30 D | lets all in |
| facebook.com | 30 D | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| homedepot.com | 30 D | lets all in |
| ibm.com | 30 D | lets all in |
| instagram.com | 30 D | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … · little text without JavaScript |
| jumia.com.ng | 30 D | blocks 1: CCBot |
| leboncoin.fr | 30 D | blocks 1: Bytespider |
| openai.com | 30 D | lets all in |
| perplexity.ai | 30 D | lets all in |
| rewe.de | 30 D | lets all in |
| takealot.com | 30 D | lets all in |
| timesofisrael.com | 30 D | lets all in |
| wayfair.com | 30 D | lets all in |
| x.ai | 30 D | lets all in |
| aliexpress.com | 28 E | blocks 6: GPTBot, ClaudeBot, Google-Extended, Amazonbot … |
| medium.com | 28 E | blocks 6: GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent … |
| tripadvisor.com | 28 E | blocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended … |
| zillow.com | 28 E | blocks 3: Applebot-Extended, Meta-ExternalAgent, Amazonbot |
| claude.ai | 25 E | blocks 4: ChatGPT-User, OAI-SearchBot, GPTBot, Google-Extended |
| ebay.com | 25 E | blocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended … |
| ebay.de | 25 E | blocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended … |
| flipkart.com | 25 E | turns readers away |
| forbes.com | 25 E | blocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended … |
| ft.com | 25 E | blocks 8: ClaudeBot, Perplexity-User, PerplexityBot, Google-Extended … |
| idealista.com | 25 E | lets all in |
| irctc.co.in | 25 E | lets all in |
| seek.com.au | 25 E | lets all in |
| vnexpress.net | 25 E | lets all in |
| ahram.org.eg | 23 E | blocks 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended … |
| bloomberg.com | 23 E | blocks 10: GPTBot, ClaudeBot, PerplexityBot, Google-Extended … |
| glassdoor.com | 23 E | blocks 7: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended … |
| kicker.de | 21 E | blocks 11: ChatGPT-User, GPTBot, Claude-SearchBot, ClaudeBot … |
| chefkoch.de | 20 E | blocks 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended … |
| apnews.com | 19 E | blocks 9: GPTBot, Claude-User, Claude-SearchBot, ClaudeBot … |
| amazon.de | 17 E | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … · little text without JavaScript |
| economist.com | 16 E | blocks 12: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| khanacademy.org | 15 E | turns readers away · little text without JavaScript |
| nation.africa | 14 E | blocks 12: ChatGPT-User, GPTBot, Claude-User, ClaudeBot … |
| nytimes.com | 14 E | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| quora.com | 14 E | blocks 9: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| realestate.com.au | 14 E | blocks 11: GPTBot, Claude-User, Claude-SearchBot, ClaudeBot … |
| reuters.com | 14 E | blocks 14: GPTBot, Claude-User, Claude-SearchBot, ClaudeBot … |
| wsj.com | 14 E | blocks 13: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User … |
| news24.com | 12 E | blocks 15: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| lowes.com | 10 E | turns readers away · little text without JavaScript |
| news.com.au | 10 E | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| zeit.de | 10 E | blocks 16: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| elpais.com | 9 E | blocks 10: Claude-User, Claude-SearchBot, ClaudeBot, Perplexity-User … |
| imdb.com | 9 E | blocks 14: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| linkedin.com | 9 E | blocks 14: ChatGPT-User, GPTBot, Claude-User, ClaudeBot … |
| adidas.com | 0 E | turns readers away |
| argos.co.uk | 0 E | turns readers away |
| booking.com | 0 E | turns readers away |
| coupang.com | 0 E | turns readers away |
| dawn.com | 0 E | turns readers away |
| elcorteingles.es | 0 E | turns readers away |
| fnac.com | 0 E | turns readers away |
| haaretz.co.il | 0 E | turns readers away |
| hm.com | 0 E | turns readers away |
| idealo.de | 0 E | turns readers away |
| macys.com | 0 E | turns readers away |
| mayoclinic.org | 0 E | turns readers away |
| mediamarkt.de | 0 E | turns readers away |
| mercedes-benz.de | 0 E | turns readers away |
| midjourney.com | 0 E | turns readers away |
| ndtv.com | 0 E | turns readers away |
| nih.gov | 0 E | turns readers away |
| oracle.com | 0 E | turns readers away |
| ox.ac.uk | 0 E | turns readers away |
| reddit.com | 0 E | blocks 18: ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User … |
| rozetka.com.ua | 0 E | turns readers away |
| sony.com | 0 E | turns readers away |
| stackoverflow.com | 0 E | turns readers away |
| subito.it | 0 E | turns readers away |
| tesco.com | 0 E | turns readers away |
| zara.com | 0 E | turns readers away |
| zomato.com | 0 E | turns readers away |
This data is free for everyone. Support the open atlas — voluntary, any amount.
Your site
Check any website — free, no account: mareliberum.dev/check
the most visited websites of the world, read again every six days from what each publishes (robots.txt, llms.txt, home page, agent card); open data, CC0. robots.txt states wishes; most large AI operators say they follow it. A site that turns automated readers away also stops most AI assistants.
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto