whisper-timestamped
An extension of OpenAI's Whisper speech-recognition model that adds accurate word-level timestamps and per-word confidence scores.
whisper-timestamped extends OpenAI's Whisper multilingual speech-recognition models to predict accurate word-level timestamps and a confidence score for each word and segment, which the original Whisper models do not provide natively. It builds on dynamic time warping applied to the model's cross-attention weights, refining an existing public technique with more accurate start and end estimation and, in most cases, no extra inference pass beyond decoding.
It is designed to be memory-efficient on long audio files and is meant to be compatible with any version of the underlying openai-whisper Python package. Optional voice-activity detection, with a choice of several methods, can run before Whisper to avoid hallucinated text on segments of pure silence, and utility functions can plot the resulting word alignment.
It is written in Python and released under the AGPL-3.0 license.