Awesome-LLMs-for-Video-Understanding
A curated survey and paper list on video understanding with large language models, covering models, training strategies, tasks, datasets and benchmarks.
Awesome-LLMs-for-Video-Understanding is the companion repository for the survey paper Video Understanding with Large Language Models: A Survey, accepted to IEEE Transactions on Circuits and Systems for Video Technology. It covers video-understanding techniques powered by large language models, including training strategies, relevant tasks, datasets, benchmarks and evaluation methods, and discusses applications of these video-LLMs across domains.
It organizes video-LLM research into a taxonomy based on video representation and LLM functionality, such as using an LLM as a summarizer, manager, text decoder, regressor or hidden layer, across combinations of video analyzers and embedders. Its most recent major update added roughly 100 additional video-LLMs and 15 new benchmarks, added a training-strategies chapter, and redesigned its figures and tables. A related follow-up work on post-training for video reasoning with large multimodal models is also linked from the repository.
Contributions of missing papers are accepted through pull requests following a documented format.