Spark NLP
An open-source natural language processing library built on Apache Spark, offering pretrained pipelines and models for tasks like tokenization, named entity recognition, sentiment analysis, translation and question answering.
Spark NLP is a natural language processing library built on top of Apache Spark. It provides annotations for machine learning pipelines that scale across a distributed environment, and ships with pretrained pipelines and models covering more than 200 languages.
Supported tasks include tokenization, word segmentation, part-of-speech tagging, embeddings, named entity recognition, dependency parsing, spell checking, text classification, sentiment analysis, machine translation, summarization, question answering, image classification, image captioning and automatic speech recognition. It includes transformer architectures such as BERT, RoBERTa, DeBERTa, XLM-RoBERTa, Llama, Mistral, Phi and Qwen2, and can import models from TensorFlow, ONNX, OpenVINO and Llama.cpp (GGUF).
It is available for Python, R and the JVM ecosystem (Java, Scala and Kotlin), extending Apache Spark natively.