Mare Liberum · open data · the state of the agent web

Which top websites block Googlebot?

0% of the 289 most visited websites read so far block Googlebot (Google) in robots.txt — 1 site. Another 32 turn all automated readers away.

Googlebot builds the index an AI search or answer engine draws from. Blocking it keeps the site out of those answers.

Updated 2026-11-19 22:51 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)

Sites that block Googlebot

reddit.com0/100 Eblocks 18 of 18 agents

Is your site one of them?

Check any website — free, no account: mareliberum.dev/check

The other agents

Bytespider 35% · ClaudeBot 33% · CCBot 33% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 28% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 17% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1%

robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto