The Atlas · the web as an agent needs it
httparchive.org for agents
The HTTP Archive — The HTTP Archive tracks how the web is built. We periodically crawl the top sites on the web and record detailed information about fetched resources, used web platform APIs and features, and execution traces of each page.
Search
The home page shows no search the Atlas can turn into a link.
Who may read it
robots.txt lets all 18 known AI agents read it.
The home page shows 369 words without JavaScript.
Sections
All Reports State of the Web State of JavaScript State of Images Loading Speed Progressive Web Apps Accessibility SEO Page Weight CrUX Capabilities Core Web Vitals Technology Report Docs 2025
Files for machines
| Sitemaps | https://httparchive.org/sitemap.xml |
| llms.txt | https://httparchive.org/llms.txt |
| Structured data | Article |
Agent-readiness 90/100 (A) · full check
Mapped 2026-11-20 · rank 15,471 in the Majestic Million · JSON · Markdown
The grammar of the Atlas
https://mareliberum.dev/site/<domain> how an agent uses any website (search link, sections, feeds, who may read) https://mareliberum.dev/find?q=<words> websites for a task, the most usable for agents first https://mareliberum.dev/brain?q=<task> pathways other agents used for the same task https://mareliberum.dev/r/<address> any page as clean Markdown https://mareliberum.dev/check/<domain> how AI agents see a website, scored
read from what the site publishes (robots.txt, the home page and the files it names) by MareLiberum-Atlas; open data, CC BY 4.0. The site can ask the Atlas to stay away: User-agent: MareLiberum-Atlas Disallow: / in its robots.txt. About the Atlas
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto