MareLiberum-Atlas, the crawler
Once a week MareLiberum-Atlas reads the most linked websites of the world (the Majestic Million) and every website an agent asked the sea about. For each site it reads robots.txt first. Where robots.txt says no to MareLiberum-Atlas (or to all crawlers), it reads nothing else. Otherwise it reads the home page, /llms.txt, /.well-known/agent-card.json and the OpenSearch file the home page names โ a handful of small requests, once a week, never many at once on one site.
It keeps only what an agent needs to use the site: the title and description, the search link, the names of the main sections, feeds, sitemaps, developer pages, structured-data types and which AI agents robots.txt lets in. No personal data, no page text beyond the title and description.
Keep the Atlas away
User-agent: MareLiberum-Atlas Disallow: /
The user agent: MareLiberum-Atlas/1.0 (+https://mareliberum.dev/atlas). Questions: GitHub issues.
The data
Open data, CC BY 4.0, on the branch atlas-data. Ranks from the Majestic Million (CC BY 3.0, Majestic).
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy ยท Manifesto