Submit a resource

Navigation

Back to directory

Scrapy

A cross-platform Python framework for crawling websites and extracting structured data at scale.

Visit website (opens in a new tab)scrapy.org
Scrapy

Scrapy is a web crawling and scraping framework for Python used to extract structured data from websites. It is cross-platform and requires Python 3.10 or later, and is maintained by Zyte along with a wider community of contributors.

The project is installed with pip and documented on a dedicated documentation site that covers how to define spiders, extract data with selectors, and run crawls. It is released under the BSD 3-Clause license and accepts external contributions through its published contributing guide.

More in Browser automation

See all 88

  • browser-session-ctl

    github.com

    Loads an unpacked Chrome extension that lets a local AI coding agent click, type and read the tabs, cookies and logins of a Chrome window you already have open.

  • Hronaut

    hronaut.dev

    A visible, persistent Electron browser that exposes durable, named agent workspaces over MCP, so a coding agent's browser session stays open and inspectable independently of any one AI conversation.

  • Prompt OS

    github.com

    A desktop AI agent for Windows that executes terminal commands, manages files, writes and runs code, browses the web and reads the screen, acting on natural-language requests rather than only answering them.

Cookies

We use necessary cookies to run the site. Measurement and marketing are optional; neither is used today. Privacy policy