Mare Liberum · open data

The Atlas: the web as an agent needs it

The web was built for people. The Atlas writes down, for every website, what an agent needs to act there without guessing: a search link with {q}, the main sections, feeds and sitemaps, developer pages, llms.txt and agent card — and whether robots.txt lets AI agents in.

346 websites on the map · 28.3% can be searched with one link · 15.3% have feeds · 18.2% have llms.txt · 13% show almost nothing without JavaScript · 25.8% turn automated readers away

Use it

https://mareliberum.dev/site/<domain>        how an agent uses any website (search link, sections, feeds, who may read)
https://mareliberum.dev/find?q=<words>      websites for a task, the most usable for agents first
https://mareliberum.dev/brain?q=<task>      pathways other agents used for the same task
https://mareliberum.dev/r/<address>        any page as clean Markdown
https://mareliberum.dev/check/<domain>       how AI agents see a website, scored

On the map

google.com youtube.com apple.com microsoft.com github.com play.google.com wikipedia.org apps.apple.com youtu.be wordpress.org wa.me en.wikipedia.org maps.google.com tiktok.com docs.google.com whatsapp.com vimeo.com policies.google.com nginx.org f5.com nginx.com amazon.com t.me drive.google.com mozilla.org europa.eu support.google.com adobe.com wordpress.com bit.ly player.vimeo.com apache.org github.io plus.google.com nih.gov spotify.com yahoo.com qq.com itunes.apple.com creativecommons.org open.spotify.com ec.europa.eu archive.org tumblr.com gravatar.com sites.google.com w3.org doi.org office.com amazonaws.com soundcloud.com accounts.google.com live.com googleblog.com zoom.us ncbi.nlm.nih.gov sourceforge.net who.int shopify.com cpanel.net wikimedia.org opera.com dropbox.com discord.com workspaceupdates.googleblog.com cnn.com mit.edu discord.gg tinyurl.com php.net pubmed.ncbi.nlm.nih.gov launchpad.net example.com bing.com harvard.edu cdc.gov cloud.microsoft un.org debian.org mp.weixin.qq.com plesk.com issuu.com gov.uk weebly.com expireddomains.com commons.wikimedia.org stripe.com outlook.com g.page arxiv.org nasa.gov calendly.com telegram.me msn.com bsky.app ietf.org gitlab.com share.google dropcatch.com amzn.to aws.amazon.com line.me substack.com openstreetmap.org statista.com cloudflare.com podcasts.apple.com mysql.com canva.com twitch.tv cnbc.com oup.com eventbrite.com chatgpt.com pmc.ncbi.nlm.nih.gov learn.microsoft.com addtoany.com samsung.com time.com techcrunch.com fr.wikipedia.org rfc-editor.org outlook.live.com chrome.google.com hubspot.com loc.gov validator.w3.org slideshare.net sagepub.com aol.com zendesk.com cornell.edu yandex.ru ca.gov outlook.office.com wp.com squarespace.com wpguardian.com wpguardian.io google.de berkeley.edu nist.gov bilibili.com de.wikipedia.org go.com search.google.com jotform.com bandcamp.com scholar.google.com edition.cnn.com instagram.com x.com

Who gets shut out

GPTBot OpenAI17.9% of sites
CCBot Common Crawl17.6% of sites
Bytespider ByteDance16.8% of sites
ClaudeBot Anthropic15.6% of sites
Google-Extended Google13% of sites
Amazonbot Amazon12.4% of sites
Meta-ExternalAgent Meta12.1% of sites
PerplexityBot Perplexity11.3% of sites
Applebot-Extended Apple10.7% of sites
ChatGPT-User OpenAI10.4% of sites

Open data, CC BY 4.0: the whole map on GitHub · /atlas.json · how the Atlas reads

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto