How AI agents see
smh.com.au
Agents can read this site — a few changes make it easier.
Checked 2026-11-19 22:50 UTC (kept for an hour) · read from what the site publishes: robots.txt, llms.txt, the home page and the agent card. robots.txt states wishes; most large AI operators say they follow it.
What counts
| ! | 6 of 11 assistants and AI search crawlers may read the site robots.txt blocks some of them — allow the ones you want to send you visitors (ChatGPT-User, OAI-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot) read from robots.txt | 14 / 25 |
| ✓ | 1139 words readable without JavaScript read from home page | 20 / 20 |
| ✓ | Structured data: WebPage read from home page | 15 / 15 |
| ✓ | Title and description present read from home page | 10 / 10 |
| ! | No llms.txt publish /llms.txt: a short plain-text guide to the site for language models read from /llms.txt | 0 / 10 |
| ! | No agent card if agents may act with you (book, buy, ask), publish /.well-known/agent-card.json (A2A) read from /.well-known/agent-card.json | 0 / 10 |
| ✓ | Sitemap found read from robots.txt / sitemap.xml | 5 / 5 |
| ✓ | Answers over HTTPS read from home page | 5 / 5 |
Who may read it — by robots.txt
Assistants that reads for a person right now:
✕ ChatGPT-User · OpenAI✓ Claude-User · Anthropic✓ Perplexity-User · Perplexity✕ DuckAssistBot · DuckDuckGo✓ MistralAI-User · Mistral
AI search that brings you into AI answers and search:
✕ OAI-SearchBot · OpenAI✓ Claude-SearchBot · Anthropic✕ PerplexityBot · Perplexity✓ Googlebot · Google✓ Bingbot · Microsoft✕ Amazonbot · Amazon
Crawlers that trains future models — your choice, not scored:
✕ GPTBot · OpenAI✕ ClaudeBot · Anthropic✕ Google-Extended · Google✕ Applebot-Extended · Apple✕ Meta-ExternalAgent · Meta✕ CCBot · Common Crawl✕ Bytespider · ByteDance
What an agent reads first
| Title | Australian Breaking News Headlines & World News Online | SMH.com.au |
| Description | Breaking news from Sydney, Australia and the world. Features the latest business, sport, entertainment, travel, lifestyle, and technology news. |
| Language | en |
| Words without JavaScript | 1,139 |
| Structured data | WebPage |
How the most visited websites compare: the state of the agent web
Show your score
A live badge for your site, README or footer. It updates when the site changes; people and agents who click it land on this result.
<a href="https://mareliberum.dev/check/smh.com.au"><img src="https://mareliberum.dev/check/smh.com.au/badge.svg" alt="How AI agents see smh.com.au"></a>
Markdown: [](https://mareliberum.dev/check/smh.com.au)
The check is free and stays free. Support the open atlas with any amount — voluntary, buys nothing.
Check another site
Want to see which AI agents actually come by? Type the site into the finder. Share this result: mareliberum.dev/check/smh.com.au
Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto