Vision Agents
An open framework for building low-latency, multi-modal AI agents that watch, listen and understand video in real time, using Stream's edge network and any model or vision provider.
Vision Agents provides building blocks for creating intelligent, low-latency video experiences powered by a developer's own choice of models, infrastructure and use cases. It targets real-time video AI, combining object-detection models such as YOLO or Roboflow with realtime multimodal models such as Gemini or OpenAI.
Built by Stream but designed to work with any video edge network, it offers native SDK methods for OpenAI, Gemini and Claude, and client SDKs for React, Android, iOS, Flutter, React Native and Unity powered by Stream's low-latency network, joining a session in about 500 milliseconds and holding audio/video latency under 30 milliseconds.
A pluggable video processor pipeline runs models such as YOLO or custom PyTorch/ONNX models before or after LLM calls, illustrated in the project's example of a golf-coaching agent that combines pose detection with a realtime Gemini model; other named use cases include drone fire detection and physical therapy coaching.