Spider
A concurrency-first web crawling and scraping engine for Rust that streams pages as they are fetched and renders JavaScript only on pages that need it. It scales from a single script to a distributed fleet, and the same engine also runs a hosted cloud service.
Spider is a crawling engine written in Rust that fetches pages concurrently, streaming them as they arrive rather than waiting for the whole crawl to finish. It runs HTTP requests first and launches headless Chrome only for pages that need JavaScript rendering, and it includes built-in proxies, retries, rate limiting and stealth options.
The same engine also runs Spider Cloud, a managed service that adds proxy rotation and automatic unblocking, so code written locally can move to hosted infrastructure with a configuration change. It ships as a Rust library, a command-line tool, packages for Node.js and Python, and an MCP server for use from AI assistants.