Mare Liberum · open data · the state of the agent web
Which top websites block Bytespider?
35% of the 289 most visited websites read so far block Bytespider (ByteDance) in robots.txt — 100 sites. Another 32 turn all automated readers away.
Bytespider collects pages to train AI models. Blocking it keeps the site out of future training data; it does not stop assistants from reading the site for a person.
Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)
Sites that block Bytespider
| weather.com | 88/100 A | blocks 3 of 18 agents |
| repubblica.it | 85/100 A | blocks 7 of 18 agents |
| github.com | 80/100 B | blocks 1 of 18 agents |
| techcrunch.com | 80/100 B | blocks 5 of 18 agents |
| temu.com | 80/100 B | blocks 1 of 18 agents |
| chatgpt.com | 78/100 B | blocks 4 of 18 agents |
| espn.com | 78/100 B | blocks 5 of 18 agents |
| srf.ch | 78/100 B | blocks 7 of 18 agents |
| abc.net.au | 75/100 B | blocks 8 of 18 agents |
| aljazeera.com | 75/100 B | blocks 5 of 18 agents |
| idnes.cz | 75/100 B | blocks 6 of 18 agents |
| tagesschau.de | 75/100 B | blocks 7 of 18 agents |
| onet.pl | 73/100 B | blocks 3 of 18 agents |
| t-online.de | 73/100 B | blocks 7 of 18 agents |
| ansa.it | 71/100 B | blocks 10 of 18 agents |
| yahoo.com | 70/100 B | blocks 7 of 18 agents |
| yle.fi | 70/100 B | blocks 8 of 18 agents |
| bild.de | 69/100 C | blocks 8 of 18 agents |
| smh.com.au | 69/100 C | blocks 12 of 18 agents |
| spiegel.de | 69/100 C | blocks 11 of 18 agents |
| healthline.com | 68/100 C | blocks 7 of 18 agents |
| sap.com | 68/100 C | blocks 4 of 18 agents |
| svt.se | 68/100 C | blocks 8 of 18 agents |
| nzz.ch | 66/100 C | blocks 13 of 18 agents |
| asahi.com | 64/100 C | blocks 14 of 18 agents |
| bbc.co.uk | 64/100 C | blocks 14 of 18 agents |
| bbc.com | 64/100 C | blocks 14 of 18 agents |
| hln.be | 64/100 C | blocks 14 of 18 agents |
| theverge.com | 64/100 C | blocks 13 of 18 agents |
| uol.com.br | 64/100 C | blocks 13 of 18 agents |
| aftonbladet.se | 63/100 C | blocks 9 of 18 agents |
| derstandard.at | 63/100 C | blocks 10 of 18 agents |
| eltiempo.com | 63/100 C | blocks 8 of 18 agents |
| emol.com | 63/100 C | blocks 7 of 18 agents |
| golem.de | 63/100 C | blocks 2 of 18 agents |
| kakaku.com | 63/100 C | blocks 2 of 18 agents |
| nos.nl | 63/100 C | blocks 10 of 18 agents |
| orf.at | 63/100 C | blocks 8 of 18 agents |
| vg.no | 63/100 C | blocks 9 of 18 agents |
| corriere.it | 62/100 C | blocks 15 of 18 agents |
| dailymail.co.uk | 60/100 C | blocks 16 of 18 agents |
| dr.dk | 60/100 C | blocks 16 of 18 agents |
| elmundo.es | 60/100 C | blocks 16 of 18 agents |
| eluniversal.com.mx | 60/100 C | blocks 15 of 18 agents |
| francetvinfo.fr | 60/100 C | blocks 16 of 18 agents |
| gazzetta.it | 60/100 C | blocks 16 of 18 agents |
| kompas.com | 60/100 C | blocks 15 of 18 agents |
| marca.com | 60/100 C | blocks 16 of 18 agents |
| nu.nl | 60/100 C | blocks 16 of 18 agents |
| publico.pt | 60/100 C | blocks 16 of 18 agents |
| rappler.com | 60/100 C | blocks 16 of 18 agents |
| telegraph.co.uk | 60/100 C | blocks 15 of 18 agents |
| lemonde.fr | 59/100 C | blocks 10 of 18 agents |
| kathimerini.gr | 58/100 C | blocks 10 of 18 agents |
| faz.net | 57/100 C | blocks 15 of 18 agents |
| netflix.com | 56/100 C | blocks 8 of 18 agents |
| theguardian.com | 54/100 C | blocks 10 of 18 agents |
| nrk.no | 52/100 C | blocks 15 of 18 agents |
| pinterest.com | 52/100 C | blocks 15 of 18 agents |
| cnn.com | 50/100 C | blocks 14 of 18 agents |
| nzherald.co.nz | 50/100 C | blocks 16 of 18 agents |
| theglobeandmail.com | 50/100 C | blocks 16 of 18 agents |
| cnbc.com | 44/100 D | blocks 14 of 18 agents |
| wired.com | 44/100 D | blocks 13 of 18 agents |
| amazon.com | 42/100 D | blocks 14 of 18 agents |
| tiktok.com | 42/100 D | blocks 15 of 18 agents |
| x.com | 40/100 D | blocks 16 of 18 agents |
| baidu.com | 37/100 D | blocks 17 of 18 agents |
| stuff.co.nz | 36/100 D | blocks 11 of 18 agents |
| canva.com | 30/100 D | blocks 6 of 18 agents |
| facebook.com | 30/100 D | blocks 16 of 18 agents |
| instagram.com | 30/100 D | blocks 16 of 18 agents |
| leboncoin.fr | 30/100 D | blocks 1 of 18 agents |
| aliexpress.com | 28/100 E | blocks 6 of 18 agents |
| medium.com | 28/100 E | blocks 6 of 18 agents |
| tripadvisor.com | 28/100 E | blocks 8 of 18 agents |
| ebay.com | 25/100 E | blocks 8 of 18 agents |
| ebay.de | 25/100 E | blocks 8 of 18 agents |
| forbes.com | 25/100 E | blocks 8 of 18 agents |
| ft.com | 25/100 E | blocks 8 of 18 agents |
| ahram.org.eg | 23/100 E | blocks 8 of 18 agents |
| bloomberg.com | 23/100 E | blocks 10 of 18 agents |
| glassdoor.com | 23/100 E | blocks 7 of 18 agents |
| kicker.de | 21/100 E | blocks 11 of 18 agents |
| chefkoch.de | 20/100 E | blocks 8 of 18 agents |
| amazon.de | 17/100 E | blocks 14 of 18 agents |
| economist.com | 16/100 E | blocks 12 of 18 agents |
| nation.africa | 14/100 E | blocks 12 of 18 agents |
| nytimes.com | 14/100 E | blocks 14 of 18 agents |
| quora.com | 14/100 E | blocks 9 of 18 agents |
| realestate.com.au | 14/100 E | blocks 11 of 18 agents |
| reuters.com | 14/100 E | blocks 14 of 18 agents |
| wsj.com | 14/100 E | blocks 13 of 18 agents |
| news24.com | 12/100 E | blocks 15 of 18 agents |
| news.com.au | 10/100 E | blocks 16 of 18 agents |
| zeit.de | 10/100 E | blocks 16 of 18 agents |
| elpais.com | 9/100 E | blocks 10 of 18 agents |
| imdb.com | 9/100 E | blocks 14 of 18 agents |
| linkedin.com | 9/100 E | blocks 14 of 18 agents |
| reddit.com | 0/100 E | blocks 18 of 18 agents |
Is your site one of them?
Check any website — free, no account: mareliberum.dev/check
The other agents
ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1% · Googlebot 0%
robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto