Submit a resource

Navigation

Back to directory

Crawl4AI

Crawl4AI is an open-source web crawler that turns any website into clean, LLM-ready Markdown for retrieval-augmented generation and AI agent pipelines, usable as a library, a self-hosted server, or a hosted API.

Visit website (opens in a new tab)crawl4ai.com
Crawl4AI

Crawl4AI runs as a Python library with an async browser-based crawler, as a Docker server and CLI for self-hosting, or as a hosted cloud service reachable through one API key that adds search, question answering, extraction and batch scraping without managing browsers or proxies.

Output is Markdown built for language models: headings, lists, tables and code blocks kept in a structure that reads well, plus filters that strip menus, footers and boilerplate. It supports sessions, proxies, cookies and custom scripts for full control, an adaptive mode that learns a site's patterns to avoid unnecessary requests, and an MCP interface so a coding agent can call scraping, search and extraction as tools.

The project is free to run yourself indefinitely; the hosted cloud option is priced per use. Python, Apache-2.0.

More in Browser automation

See all 88

  • browser-session-ctl

    github.com

    Loads an unpacked Chrome extension that lets a local AI coding agent click, type and read the tabs, cookies and logins of a Chrome window you already have open.

  • Hronaut

    hronaut.dev

    A visible, persistent Electron browser that exposes durable, named agent workspaces over MCP, so a coding agent's browser session stays open and inspectable independently of any one AI conversation.

  • Prompt OS

    github.com

    A desktop AI agent for Windows that executes terminal commands, manages files, writes and runs code, browses the web and reads the screen, acting on natural-language requests rather than only answering them.

Cookies

We use necessary cookies to run the site. Measurement and marketing are optional; neither is used today. Privacy policy