OpenCrawling
The reference Java and Spring implementation of the Open Ingestion Standard, a vendor-neutral enterprise data-integration platform that orchestrates connectors, embedding, and vector search output using modern Java concurrency features.
OpenCrawling implements the Open Ingestion Standard in Java and Spring, providing a secure, decoupled, vendor-neutral platform for orchestrating data flows from repository connectors such as SharePoint, S3, and filesystem sources through to vector search output, using Java 25 features such as structured concurrency and virtual threads alongside Spring Boot and Spring AI. Its architecture separates a stateless embedding microservice from transformation connectors so each can scale and be configured independently.
An administration dashboard shows real-time telemetry on job success rates, Kafka queue load, active crawling threads, and indexing speed, schedules and monitors ingestion jobs, and manages connector endpoints and credentials along with embedding model mappings and chunking settings for targets such as Ollama or OpenAI. A separate terminal-native command-line interface, built with Picocli and JLine, supports DevSecOps automation, CI/CD integration, connector health checks, and offline validation of ingestion configurations against the standard's schema, and it can be run as a built JAR or a Docker container.