Scrapy MCP server
View on GitHub ↗This is an MCP server that can connect to live Scrapy crawls to inspect and control them.
Attach an MCP-capable agent (e.g. Claude Code) to a running Scrapy crawl and inspect, debug, and steer it from the inside — by running async Python inside the live crawl process. It's a structured, agent-oriented successor to Scrapy's telnet console: the agent writes a snippet, it runs on the crawl's own event loop, and the output comes back.
Once attached, the agent can answer the operational questions you'd otherwise dig for by hand:
- How much has it scraped, and what's the request error rate? - What's downloading right now, and from which domains? - Is it making progress or stuck — and if stuck, why? - What settings / middlewares / pipelines are actually running? - Why is it slow? (download delay / concurrency / autothrottle)
It can also explore fixes read-only — e.g. trying candidate selectors against a response held in memory and diffing the results — and, more experimentally, patch the running spider.
Two pieces are installed separately (but can be installed in the same Python venv if desired):
- The Scrapy RemoteControl extension: ships with Scrapy 2.19.0 and later and is enabled by default. While the crawl is running it listens on a random localhost port for HTTP requests. You need to know the port number and the auth token to connect to it, which you can get from a job file created by the extension in the user profile directory. - MCP server: this package. It's a standard stdio MCP server your agent (e.g. Claude Code) launches. It discovers jobs by reading job files and
From the project README.
Add the radar badge to your README — it shows your project was picked up by MCP Radar and links to this page:
[](https://mcp.liqiwa.com/s/scrapy--scrapy-mcp-official.html)
Stop your AI from making things up — it proposes, deterministic tools decide, every claim checked against ground truth with evidence. Grounded facts and context survive resets. Reverse engineering is
zhuyansen/new-term-radarDaily radar for new AI terms: names of new models, products and repos, caught on X, Hugging Face, GitHub and sitemaps before keyword tools list them.
yzg1226/mcd-party-officer🍟 麦当劳门店派对成团调度 Skill — 把派对从「策划」推进到「办得成」：成团概率雷达 / 包场vs拼团 / 择日黄历 / 儿童营养指数 / 招募卡与邀请函
fellowgeek/mcp-memoryAn OKF-backed Model Context Protocol (MCP) server delivering persistent long-term memory and SQLite FTS5 search for AI agents.
hirotomasato/yowesGenerate realistic teacher documents (ID cards, licenses, letters) for 13 countries via MCP.
hujjun/mcd-saver-strategist麦当劳省钱点餐助手（麦门精算师）｜麦当劳优惠券组合优化 Skill：拉取麦当劳菜单与全部可用优惠券，算出最省的点餐组合，巨无霸套餐 ¥47 → ¥21.9（省 53%）。McDonalds MCP（麦当劳 MCP）coupon optimizer · McDonalds China MCP Server · Agent Skill for WorkBuddy · 麦当劳程序员创意开发大赛参赛作品
The top new MCP servers of the week, every Monday. No spam, unsubscribe anytime.