How AI agents see
mit.edu
Agents can read this site — a few changes make it easier.
Checked 2026-11-19 22:32 UTC (kept for an hour) · read from what the site publishes: robots.txt, llms.txt, the home page and the agent card. robots.txt states wishes; most large AI operators say they follow it.
What counts
| ✓ | 11 of 11 assistants and AI search crawlers may read the site read from robots.txt | 25 / 25 |
| ✓ | 365 words readable without JavaScript read from home page | 20 / 20 |
| ! | No structured data (JSON-LD) describe what you offer as schema.org JSON-LD (Organization, Product, Offer, Article …) — agents read it first read from home page | 5 / 15 |
| ✓ | Title and description present read from home page | 10 / 10 |
| ! | No llms.txt publish /llms.txt: a short plain-text guide to the site for language models read from /llms.txt | 0 / 10 |
| ! | No agent card if agents may act with you (book, buy, ask), publish /.well-known/agent-card.json (A2A) read from /.well-known/agent-card.json | 0 / 10 |
| ! | No sitemap found in robots.txt, /sitemap.xml or /sitemap_index.xml list your pages in /sitemap.xml and name it in robots.txt read from robots.txt / sitemap.xml | 0 / 5 |
| ✓ | Answers over HTTPS read from home page | 5 / 5 |
Who may read it — by robots.txt
Assistants that reads for a person right now:
✓ ChatGPT-User · OpenAI✓ Claude-User · Anthropic✓ Perplexity-User · Perplexity✓ DuckAssistBot · DuckDuckGo✓ MistralAI-User · Mistral
AI search that brings you into AI answers and search:
✓ OAI-SearchBot · OpenAI✓ Claude-SearchBot · Anthropic✓ PerplexityBot · Perplexity✓ Googlebot · Google✓ Bingbot · Microsoft✓ Amazonbot · Amazon
Crawlers that trains future models — your choice, not scored:
✓ GPTBot · OpenAI✓ ClaudeBot · Anthropic✓ Google-Extended · Google✓ Applebot-Extended · Apple✓ Meta-ExternalAgent · Meta✓ CCBot · Common Crawl✓ Bytespider · ByteDance
What an agent reads first
| Title | MIT - Massachusetts Institute of Technology |
| Description | The mission of MIT is to advance knowledge and educate students in science, technology and other areas of scholarship that will best serve the nation and the world in the 21st century. |
| Language | en |
| Words without JavaScript | 365 |
| Structured data | — |
How the most visited websites compare: the state of the agent web
Show your score
A live badge for your site, README or footer. It updates when the site changes; people and agents who click it land on this result.
<a href="https://mareliberum.dev/check/mit.edu"><img src="https://mareliberum.dev/check/mit.edu/badge.svg" alt="How AI agents see mit.edu"></a>
Markdown: [](https://mareliberum.dev/check/mit.edu)
The check is free and stays free. Support the open atlas with any amount — voluntary, buys nothing.
Check another site
Want to see which AI agents actually come by? Type the site into the finder. Share this result: mareliberum.dev/check/mit.edu
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto