# Trafilatura

A Python package and command-line tool that crawls, downloads, and extracts the main text, metadata, and comments from web pages into CSV, JSON, HTML, Markdown, or XML.

Trafilatura is a Python package and command-line tool for gathering text from the web and turning raw HTML into structured data. It handles web crawling, downloading, scraping, and extraction of main text, metadata, and comments, without requiring a database, and it can process sitemaps and ATOM, JSON, or RSS feeds as well as previously downloaded HTML files or parsed HTML trees.

Its extractor focuses on the actual content, avoiding boilerplate such as headers and footers, using its own rule-based extractor with jusText and readability-lxml as fallbacks. It also extracts metadata such as title, author, date, site name, categories, and tags, preserves formatting like paragraphs, lists, quotes, and code, and optionally pulls out comments, links, images, and tables. Output can be written as plain text, Markdown, CSV, JSON, HTML, XML, or XML-TEI.

Website: https://trafilatura.readthedocs.io