The Atlas · the web as an agent needs it
commoncrawl.org for agents
Common Crawl - Open Repository of Web Crawl Data — We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
Search commoncrawl.org with one link
https://commoncrawl.org/search?query={q}Put the words where {q} stands. From the site's search form. Read the result as Markdown: https://mareliberum.dev/r/https://commoncrawl.org/search?query=example
Who may read it
robots.txt lets all 18 known AI agents read it.
The home page shows 411 words without JavaScript.
Sections
Overview CDXJ Index URL Index Web Graphs Latest Crawl Errata Get Started Blog CCBot Opt-Out Ledger FAQ Research Papers Collaborators About
Files for machines
| Sitemaps | https://commoncrawl.org/sitemap.xml |
Agent-readiness 65/100 (C) · full check
Mapped 2026-11-20 · rank 19,435 in the Majestic Million · JSON · Markdown
The grammar of the Atlas
https://mareliberum.dev/site/<domain> how an agent uses any website (search link, sections, feeds, who may read) https://mareliberum.dev/find?q=<words> websites for a task, the most usable for agents first https://mareliberum.dev/brain?q=<task> pathways other agents used for the same task https://mareliberum.dev/r/<address> any page as clean Markdown https://mareliberum.dev/check/<domain> how AI agents see a website, scored
read from what the site publishes (robots.txt, the home page and the files it names) by MareLiberum-Atlas; open data, CC BY 4.0. The site can ask the Atlas to stay away: User-agent: MareLiberum-Atlas Disallow: / in its robots.txt. About the Atlas
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto