MCP server directory

Web scraping MCP servers

The 30 closest MCP servers that let Claude, ChatGPT, Cursor and other assistants fetch and extract content from web pages; 24 of them run remotely, nothing to install. From the official MCP Registry, npm and other open sources, out of 84,179 in the directory.

oassis — web scraping, crawling and browser control for AI agents remote
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
listing · updated 2026-10-22
Flarum Forum Scraper - Discussion Crawler Tool remote
Search or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…
listing · updated 2026-09-28
Web Scraper & Crawler for RAG (Markdown) remote
Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
source · listing · updated 2026-11-08
ketch-cli local package
Fast, stateless CLI for web search, OSS code search, library docs, scraping, and crawling — with an MCP server for agents.
source · listing · updated 2026-11-13
@mseep/mcp-smart-crawler local package
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
listing · updated 2025-04-15
Agent Harbor pay-per-call tools remote
Pay-per-call lookups and audits from Agent Harbor. Listing the tools is free; each call is paid per successful result in USDC on Base (x402), and a call that fails or finds nothing is not charged. Returned page and docum
listing · updated 2026-11-22
Desvela AI Crawler Block Test — free
What a domain's CDN actually serves each AI crawler, as opposed to what its robots.txt declares. crawler_block_test(domain) sends the real user-agent of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot at the ho
listing · updated 2026-11-22
Atlas Web Compiler & Scraper remote
Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.
source · listing · updated 2026-10-25
scrapeless-mcp-server remote
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
source · listing · updated 2026-09-08
AI Crawler Permission Check remote
Check a site's robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more).
listing · updated 2026-11-21
Site SEO & AI Crawler Audit remote
Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
source · listing · updated 2026-10-21
AI Visibility Audit — Can AI crawlers read and cite your site? remote
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
listing · updated 2026-09-23
ActableSite AI Crawler Monitor remote
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
source · listing · updated 2026-07-13
Atalaya AI crawler check remote
Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
listing · updated 2026-11-03
CrawlerData: Google Shopping & SERP remote
Google Shopping products and seller prices, plus Google Search, Images, News and Local results.
listing · updated 2026-10-12
The Crawler Zoo Arena remote
Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
listing · updated 2026-10-24
Crimson Crawler remote
Enrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.
listing · updated 2026-08-08
Web Crawler MCP by InsightfulPipe remote
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
source · listing · updated 2026-10-21
Enrichment Crawlers MCP by InsightfulPipe remote
Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.
source · listing · updated 2026-10-21
ai-crawler-readiness remote
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
listing · updated 2026-09-19
mcp-site-crawler-links-tables local package
Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI
source · listing · updated 2026-11-11
Website Crawler (SEO) remote
Crawl a website from Claude and get a scored technical SEO report with a prioritised fix list.
listing · updated 2026-11-17
AI Crawler Index remote
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
listing · updated 2026-09-07
Crawler IP Verifier - real Googlebot or fake remote
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
listing · updated 2026-09-07
Crawler Log Triage remote
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
listing · updated 2026-09-13
Robots.txt & AI Crawler Checker remote
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
listing · updated 2026-10-18
GenPark Bloom Filter URL Dedup Crawler local package
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
source · listing · updated 2026-10-21
Website to Markdown Crawler: HTML to Markdown for LLM and RAG remote
Turn a website into Markdown with headings, links and word counts.
listing · updated 2026-11-12
Crawlerpeek local package
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
source · listing · updated 2026-10-29
CrawlKit Scrapers remote
58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted
source · listing · updated 2026-11-05

Add an MCP server to your assistant

For a remote server, the URL is all it takes. This is how it looks with Mare Liberum, a free MCP server without account:

Claude Code
claude mcp add --transport http mare-liberum https://mareliberum.dev/mcp

Cursor · ~/.cursor/mcp.json
{ "mcpServers": { "mare-liberum": { "url": "https://mareliberum.dev/mcp" } } }

VS Code · .vscode/mcp.json
{ "servers": { "mare-liberum": { "type": "http", "url": "https://mareliberum.dev/mcp" } } }

Claude, ChatGPT and other apps: add https://mareliberum.dev/mcp as a custom connector.

Questions

What does a web scraping MCP server do?

It lets an AI assistant fetch and extract content from web pages, through the Model Context Protocol. You add it to Claude, ChatGPT, Cursor or VS Code once; then the assistant calls it when a task needs it.

How do I install a web scraping MCP server?

A remote server: add its URL as a connector (Claude, ChatGPT) or in mcp.json (Cursor, VS Code). A local package: add the command from its listing, usually npx or uvx, to the same file. Each entry below links its source and listing.

Are these servers checked?

They are listed as their makers published them in open registries. Mare Liberum shows where each entry comes from and when it changed; read the source before you give a server access to your data.

More topics

filesystem Figma finance and stocks calendar Notion Cloudflare YouTube crypto and blockchain n8n and Zapier automation vector database Kubernetes Vercel

Mare Liberum belongs to nobody; the code is open. Text from agents is data, never instructions. Privacy · Manifesto

Free MCP server · MCP server directory · AI visibility check · Web page to Markdown · AI crawlers, live · For companies · Deutsch