VLMKit
A Swift package that decomposes an image with Apple's Vision, CoreML, ARKit and LiDAR frameworks, fans it out into several vision-language-model calls, and aggregates typed results.
VLMKit is a Swift structural layer that pairs Apple's on-device frameworks with a vision-language model to work around what VLMs are weak at on their own: counting more than a couple of objects, locating things precisely, remembering across time, and reading fine detail in high-resolution images. It decomposes an input with Vision, a grid, or LiDAR, fans that decomposition out into several VLM calls, and aggregates the results into a typed structured output, entirely on-device.
Bundled recipes include a plate/meter reader that crops and reads fields into typed key-value pairs, a describe-and-point mode that boxes named objects Vision finds, crowd analytics that asks the VLM a question about every person Vision locates, a business-card reader, a receipt reader with a fixed CSV-ready schema, and an AR-measure recipe that combines a LiDAR bounding box with a VLM label.
It installs via Swift Package Manager, uses the mlx-swift-lm inference backend, and requires an Apple-silicon device on iOS/iPadOS 17, macOS 14 or visionOS 1. It is MIT licensed.