SpeechColab

non-profit

SpeechColab

Activity Feed Request to join this org

AI & ML interests

Machine Learning for Audio/Speech

Recent Activity

yfyeung authored a paper 24 days ago

k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning

yfyeung authored a paper 3 months ago

LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization

yfyeung authored a paper 4 months ago

Zipformer: A faster and better encoder for automatic speech recognition

View all activity

speechcolab's activity

yfyeung

authored a paper 24 days ago

k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning

Paper • 2411.17100 • Published 26 days ago

yfyeung

authored a paper 3 months ago

LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization

Paper • 2409.00819 • Published Sep 1

yfyeung

authored 8 papers 4 months ago

Zipformer: A faster and better encoder for automatic speech recognition

Paper • 2310.11230 • Published Oct 17, 2023

VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech

Paper • 2401.14321 • Published Jan 25

GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement

Paper • 2406.11546 • Published Jun 17

Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS

Paper • 2309.07377 • Published Sep 14, 2023

Blank-regularized CTC for Frame Skipping in Neural Transducer

Paper • 2305.11558 • Published May 19, 2023

jimbozhang

authored 2 papers 7 months ago

GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio

Paper • 2106.06909 • Published Jun 13, 2021

speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment

Paper • 2104.01378 • Published Apr 3, 2021

jimbozhang

authored a paper 10 months ago

CED: Consistent ensemble distillation for audio tagging

Paper • 2308.11957 • Published Aug 23, 2023

sanchit-gandhi

posted an update 10 months ago

Post

Why does returning timestamps help Whisper reduce hallucinations? 🧐

Empirically, most practitioners have found that setting return_timestamps=True helps reduce hallucinations, particularly when doing long-form evaluation with Transformers’ “chunked” algorithm.

But why does this work?..

My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:

The cat sat on the on the on the mat.

Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:

<|0.00|> The cat sat on the on the on the mat.<|5.02|>

However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:

<|0.00|> The cat sat on the mat.<|5.02|>

In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356) → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).

Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?