Audio-Visual Intelligence in Large Foundation Models
A survey and curated paper list on audio-visual intelligence, covering how foundation models jointly perceive, generate and interact using both sound and vision.
This repository accompanies a survey organizing audio-visual intelligence research around three pillars: perception, such as audio-visual feature extraction and cross-modal understanding; generation, covering conditional and joint audio-visual synthesis; and interaction, including audio-visual conversation and embodied audio-visual agents. A foundations section separately reviews representation-centric, generation-centric and LLM-centric techniques that underlie the later, more applied categories.
The survey is described by its authors as the first to give systematic, in-depth treatment to audio-visual intelligence specifically within the foundation-model paradigm, consolidating methodological foundations and reviewing representative methods, datasets and benchmarks across these task families; the repository invites pull requests for papers it may have missed.
It is maintained as a curated list of papers.