Mare Liberum · open data
The Atlas: the web as an agent needs it
The web was built for people. The Atlas writes down, for every website, what an agent needs to act there without guessing: a search link with {q}, the main sections, feeds and sitemaps, developer pages, llms.txt and agent card — and whether robots.txt lets AI agents in.
346 websites on the map · 28.3% can be searched with one link · 15.3% have feeds · 18.2% have llms.txt · 13% show almost nothing without JavaScript · 25.8% turn automated readers away
Use it
https://mareliberum.dev/site/<domain> how an agent uses any website (search link, sections, feeds, who may read) https://mareliberum.dev/find?q=<words> websites for a task, the most usable for agents first https://mareliberum.dev/brain?q=<task> pathways other agents used for the same task https://mareliberum.dev/r/<address> any page as clean Markdown https://mareliberum.dev/check/<domain> how AI agents see a website, scored
On the map
google.com youtube.com apple.com microsoft.com github.com play.google.com wikipedia.org apps.apple.com youtu.be wordpress.org wa.me en.wikipedia.org maps.google.com tiktok.com docs.google.com whatsapp.com vimeo.com policies.google.com nginx.org f5.com nginx.com amazon.com t.me drive.google.com mozilla.org europa.eu support.google.com adobe.com wordpress.com bit.ly player.vimeo.com apache.org github.io plus.google.com nih.gov spotify.com yahoo.com qq.com itunes.apple.com creativecommons.org open.spotify.com ec.europa.eu archive.org tumblr.com gravatar.com sites.google.com w3.org doi.org office.com amazonaws.com soundcloud.com accounts.google.com live.com googleblog.com zoom.us ncbi.nlm.nih.gov sourceforge.net who.int shopify.com cpanel.net wikimedia.org opera.com dropbox.com discord.com workspaceupdates.googleblog.com cnn.com mit.edu discord.gg tinyurl.com php.net pubmed.ncbi.nlm.nih.gov launchpad.net example.com bing.com harvard.edu cdc.gov cloud.microsoft un.org debian.org mp.weixin.qq.com plesk.com issuu.com gov.uk weebly.com expireddomains.com commons.wikimedia.org stripe.com outlook.com g.page arxiv.org nasa.gov calendly.com telegram.me msn.com bsky.app ietf.org gitlab.com share.google dropcatch.com amzn.to aws.amazon.com line.me substack.com openstreetmap.org statista.com cloudflare.com podcasts.apple.com mysql.com canva.com twitch.tv cnbc.com oup.com eventbrite.com chatgpt.com pmc.ncbi.nlm.nih.gov learn.microsoft.com addtoany.com samsung.com time.com techcrunch.com fr.wikipedia.org rfc-editor.org outlook.live.com chrome.google.com hubspot.com loc.gov validator.w3.org slideshare.net sagepub.com aol.com zendesk.com cornell.edu yandex.ru ca.gov outlook.office.com wp.com squarespace.com wpguardian.com wpguardian.io google.de berkeley.edu nist.gov bilibili.com de.wikipedia.org go.com search.google.com jotform.com bandcamp.com scholar.google.com edition.cnn.com instagram.com x.com
Who gets shut out
| GPTBot OpenAI | 17.9% of sites |
| CCBot Common Crawl | 17.6% of sites |
| Bytespider ByteDance | 16.8% of sites |
| ClaudeBot Anthropic | 15.6% of sites |
| Google-Extended Google | 13% of sites |
| Amazonbot Amazon | 12.4% of sites |
| Meta-ExternalAgent Meta | 12.1% of sites |
| PerplexityBot Perplexity | 11.3% of sites |
| Applebot-Extended Apple | 10.7% of sites |
| ChatGPT-User OpenAI | 10.4% of sites |
Open data, CC BY 4.0: the whole map on GitHub · /atlas.json · how the Atlas reads
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto