Retrieval-Augmented Manipulation
A research implementation of a retrieval-augmented framework that gives a vision-language model spatial awareness for object-centric robot manipulation.
This project implements a retrieval-augmented framework that gives a vision-language model spatial awareness for object-centric robot manipulation, combining a retrieval component with VLM-based grounding, built on GroundingDINO, SAM2, DINOv2 and VGGT checkpoints, to help a robot arm locate and reason about objects in a scene.
It supports real hardware through a Fairino robot-arm SDK and ZED or RealSense camera bindings, configured through one file covering local paths, devices, robot IP, camera calibration and model checkpoints. A BlenderProc-based pipeline renders synthetic training scenes, including a chosen subset of objects, in BOP format from CAD models and texture libraries, feeding a training entry point for the retrieval-augmented-manipulation model on BOP-format data.
It was tested with Ubuntu 22.04, Python 3.10 and CUDA 12.8, and several dependencies, including the robot and camera SDKs and the segmentation and grounding checkpoints, must be installed separately from the main requirements file.