Submit a resource

Navigation

Back to directory

Trafilatura

A Python package and command-line tool that crawls, downloads, and extracts the main text, metadata, and comments from web pages into CSV, JSON, HTML, Markdown, or XML.

Visit website (opens in a new tab)trafilatura.readthedocs.io
Trafilatura

Trafilatura is a Python package and command-line tool for gathering text from the web and turning raw HTML into structured data. It handles web crawling, downloading, scraping, and extraction of main text, metadata, and comments, without requiring a database, and it can process sitemaps and ATOM, JSON, or RSS feeds as well as previously downloaded HTML files or parsed HTML trees.

Its extractor focuses on the actual content, avoiding boilerplate such as headers and footers, using its own rule-based extractor with jusText and readability-lxml as fallbacks. It also extracts metadata such as title, author, date, site name, categories, and tags, preserves formatting like paragraphs, lists, quotes, and code, and optionally pulls out comments, links, images, and tables. Output can be written as plain text, Markdown, CSV, JSON, HTML, XML, or XML-TEI.

More in Browser automation

See all 88

  • browser-session-ctl

    github.com

    Loads an unpacked Chrome extension that lets a local AI coding agent click, type and read the tabs, cookies and logins of a Chrome window you already have open.

  • Hronaut

    hronaut.dev

    A visible, persistent Electron browser that exposes durable, named agent workspaces over MCP, so a coding agent's browser session stays open and inspectable independently of any one AI conversation.

  • Prompt OS

    github.com

    A desktop AI agent for Windows that executes terminal commands, manages files, writes and runs code, browses the web and reads the screen, acting on natural-language requests rather than only answering them.

Cookies

We use necessary cookies to run the site. Measurement and marketing are optional; neither is used today. Privacy policy