The Atlas · the web as an agent needs it

commoncrawl.org for agents

Common Crawl - Open Repository of Web Crawl Data — We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.

Search commoncrawl.org with one link

https://commoncrawl.org/search?query={q}

Put the words where {q} stands. From the site's search form. Read the result as Markdown: https://mareliberum.dev/r/https://commoncrawl.org/search?query=example

Who may read it

robots.txt lets all 18 known AI agents read it.

The home page shows 411 words without JavaScript.

Sections

Overview CDXJ Index URL Index Web Graphs Latest Crawl Errata Get Started Blog CCBot Opt-Out Ledger FAQ Research Papers Collaborators About

Files for machines

Sitemapshttps://commoncrawl.org/sitemap.xml

Agent-readiness 65/100 (C) · full check

Mapped 2026-11-20 · rank 19,435 in the Majestic Million · JSON · Markdown

The grammar of the Atlas

https://mareliberum.dev/site/<domain>        how an agent uses any website (search link, sections, feeds, who may read)
https://mareliberum.dev/find?q=<words>      websites for a task, the most usable for agents first
https://mareliberum.dev/brain?q=<task>      pathways other agents used for the same task
https://mareliberum.dev/r/<address>        any page as clean Markdown
https://mareliberum.dev/check/<domain>       how AI agents see a website, scored

read from what the site publishes (robots.txt, the home page and the files it names) by MareLiberum-Atlas; open data, CC BY 4.0. The site can ask the Atlas to stay away: User-agent: MareLiberum-Atlas Disallow: / in its robots.txt. About the Atlas

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto