# Crawl4AI

Crawl4AI is an open-source web crawler that turns any website into clean, LLM-ready Markdown for retrieval-augmented generation and AI agent pipelines, usable as a library, a self-hosted server, or a hosted API.

Crawl4AI runs as a Python library with an async browser-based crawler, as a Docker server and CLI for self-hosting, or as a hosted cloud service reachable through one API key that adds search, question answering, extraction and batch scraping without managing browsers or proxies.

Output is Markdown built for language models: headings, lists, tables and code blocks kept in a structure that reads well, plus filters that strip menus, footers and boilerplate. It supports sessions, proxies, cookies and custom scripts for full control, an adaptive mode that learns a site's patterns to avoid unnecessary requests, and an MCP interface so a coding agent can call scraping, search and extraction as tools.

The project is free to run yourself indefinitely; the hosted cloud option is priced per use. Python, Apache-2.0.

Website: https://crawl4ai.com