Context prompts in Whisper-tiny: testing terminology gains versus added words
I’m preparing a context-prompt comparison for openai/whisper-tiny in our Integrated Model Environment (IME), Sinaps Studio. The setup has an optional audio-context field, and I want to evaluate terminology gains without mistaking prompt-driven insertions for recognition.
The model card warns that transcripts can contain words not spoken in the audio. This is a test-design question, not a claim that we reproduced a particular failure.
My proposed comparison is the same clips with:
- No context prompt.
- A short vocabulary prompt containing relevant terms, without the reference transcript.
- An unrelated vocabulary prompt as a negative control.
I would hold the model revision, preprocessing, language/task and decoding settings fixed, then report WER and insertions separately from recognition of the target terms. Silence-only clips would be scored separately, rather than assigning a WER to an empty reference.
Has anyone used a public English or Portuguese test set for this kind of comparison? What additional controls would help separate vocabulary guidance from added text?
The screenshot shows configuration only; no transcription result or accuracy measurement is shown.