OpenOmni
An open-source omnimodal large language model that combines image, speech and text understanding with real-time, emotionally expressive speech generation.
OpenOmni is an end-to-end, fully open-source omnimodal large language model that incorporates image, speech and text together. It generates speech through a language-bridging approach with text-guided speech generation, designed to train with limited omni-modal data and GPU memory.
It supports two real-time emotional speech generation modes, a CTC mode and an autoregressive mode, letting a user choose a balance between generation speed and quality. Its design has been applied to downstream tasks such as speech-embodied navigation and multi-role speech dialogue.
It is implemented in Python, installed from source with Conda, and the paper, code, models and datasets are released alongside the repository.