Mare Liberum · open data · the state of the agent web
Which top websites block Bingbot?
1% of the 289 most visited websites read so far block Bingbot (Microsoft) in robots.txt — 2 sites. Another 32 turn all automated readers away.
Bingbot builds the index an AI search or answer engine draws from. Blocking it keeps the site out of those answers.
Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)
Sites that block Bingbot
| baidu.com | 37/100 D | blocks 17 of 18 agents |
| reddit.com | 0/100 E | blocks 18 of 18 agents |
Is your site one of them?
Check any website — free, no account: mareliberum.dev/check
The other agents
Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Googlebot 0%
robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto