ScaleDP
An open-source Apache Spark extension for large-scale document processing that combines OCR, LLMs, and vision-language models to extract structured data from PDFs and images at scale.
ScaleDP is an open-source library that adds AI and ML-based document processing to Apache Spark. It loads PDF documents and images into a Spark DataFrame, extracts text and images from them, and uses LLMs and vision-language models such as GPT, Gemini, and Groq-hosted models through an OpenAI-compatible API to perform zero-shot structured-data extraction, OCR of images and PDFs, and named-entity recognition, alongside classical NLP models from the Hugging Face Hub for the same tasks.
It also supports several open-source OCR engines, including Tesseract, EasyOCR, Surya, and DocTR, as well as YOLO-based object and text detection on images. Processing can run as a REST API service without a Spark session for lower-latency use, or in a streaming mode for real-time documents.
It installs with pip or runs from a Docker image that includes a Jupyter Notebook environment for the provided quickstart.