# commoncrawl.org — for agents

**Common Crawl - Open Repository of Web Crawl Data** — We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.

**Search:** `https://commoncrawl.org/search?query={q}` (replace {q} with the words; from the site's search form)
**Readable:** 411 words without JavaScript
**Agents:** none of the 18 known agents is blocked

## Sections
- [Overview](https://commoncrawl.org/overview)
- [CDXJ Index](https://commoncrawl.org/cdxj-index)
- [URL Index](https://commoncrawl.org/url-index)
- [Web Graphs](https://commoncrawl.org/web-graphs)
- [Latest Crawl](https://commoncrawl.org/latest-crawl)
- [Errata](https://commoncrawl.org/errata)
- [Get Started](https://commoncrawl.org/get-started)
- [Blog](https://commoncrawl.org/blog)
- [CCBot](https://commoncrawl.org/ccbot)
- [Opt-Out Ledger](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry)
- [FAQ](https://commoncrawl.org/faq)
- [Research Papers](https://commoncrawl.org/research-papers)
- [Collaborators](https://commoncrawl.org/collaborators)
- [About](https://commoncrawl.org/about)

## Files
- Sitemaps: https://commoncrawl.org/sitemap.xml

Agent-readiness 65/100 (C). Mapped 2026-11-20 · rank 19435.

Read any page: https://mareliberum.dev/r/https://commoncrawl.org/ · Full check: https://mareliberum.dev/check/commoncrawl.org

_read from what the site publishes (robots.txt, the home page and the files it names) by MareLiberum-Atlas; open data, CC BY 4.0_