openai

/

whisper-large-v2

like 1.56k

Hi everyone,

I have been using the whisper-large-v2 model to transcribe audio files that are about 10 to 20 minutes long. I have tried many different techniques to improve the accuracy of the transcription, but so far, nothing has worked.

One of the main issues I am facing is that there is repeated translation information between chunks, and some translation information is missing altogether. I have tried normalizing the audio to [-1,1], as well as tuning hyperparameters such as chunk_length_s and stride_length_s. However, I have found that the optimal parameters vary from file to file, and there is no generalization.

I am using the following environment:

• whisper-large-v2
• torch 1.10.0+cu111
• transformers 4.28.1 (latest)

Here is the code I am using:

pipe = pipeline(
    task="automatic-speech-recognition",
    model=MODEL_NAME,
    device='cuda:0',
    generate_kwargs={"task": "transcribe"},
)
sound = AudioSegment.from_file("sample.mp3", format="mp3")
sound.export("sample.wav", format="wav")
y, sr = librosa.load("sample.wav", sr=None)
y_resampled = librosa.resample(y, orig_sr=sr, target_sr=16000)

outputs = pipe(y_resampled, return_timestamps=True, generate_kwargs={"task": "transcribe", "language": "<|zh|>"}, chunk_length_s=30, stride_length_s=5, batch_size=16, max_new_tokens=512)
print(outputs["text"])

Overall, I feel that the performance of the model is not good enough for me to use it formally. I would appreciate any suggestions or advice on how to improve the accuracy of the transcription.

Thank you.