Improving Speech Recognition with Jargon Injection (2024)

Abstract: This paper introduces a new method that improves the performance of Automatic speech recognition (ASR) engines, e.g., Whisper in practical cases. Different from prior methods that usually require both speech data and its transcription for decoding, our method only uses jargon as the context for decoding. To do that, the method first represents the jargon in a trie tree structure for efficient storing and traversing. The method next forces the decoding of Whisper to more focus on the jargon by adjusting the probability of generated tokens with the use of the trie tree. To further improve the performance, the method utilizes the prompting method that uses the jargon as the context. Final tokens are generated based on the combination of prompting and decoding. Experimental results on Japanese and English datasets show that the proposed method helps to improve the performance of Whisper, specially for domain-specific data. The method is simple but effective and can be deployed to any encoder-decoder ASR engines in actual cases. The code and data are also accessible (https://shorturl.at/nMsaY).

## introduction automatic speech recognition (asr) is the task of automatically transcribing input audio to output text @xcite . the output of asr systems can be used in several applications such as intelligent personal assistants @xcite , voice searches @xcite , or meeting analyses @xcite @xcite @xcite . recently, the performance of end-to-end asr models has been improved by several approaches such as connectionist temporal classification @xcite , recurrent neural network transducer @xcite , attention-based encoderdecoder @xcite @xcite 1 https://shorturl.at/yibur @xcite with strong asr engines @xcite . among those, whisper has shown strong performance for speech recognition @xcite . it was trained with 680,000 hours of labeled audio data with multitasking and multilingual learning. strong asr engines such as whisper have achieved promising results in english, yet, we observe the decent accuracy of asr engines applied to actual business, especially for low-resource languages, e.g., japanese. to fill the gap, there are two possible solutions for domain adaptation. the first well-known solution is to continuously fine-tune asr engines with domain-specific data @xcite . however, creating training corpora (including speech and text) data is a non-trivial task that is time-consuming and laborexpensive. in many cases, the creation requires domain experts, especially for narrow specific domains, e.g., high-pressure gas incidents. also, finetuning is a complex process that requires skilled practitioners @xcite . the second solution is to consider domain-specific data as a context and inject the context into the decoding phase of asr engines @xcite @xcite @xcite . among them, biasing methods are simple and potential to inject a context into the asr process. however, these methods are usually used with hybrid asr @xcite or ctc end-to-end models @xcite which are behind the performance of encoder-decoder asr models such as whisper. this paper addresses the problem of improving the performance of ars engines by using jargon. the problem comes from the fact that in practical cases, only jargon (domain-specific terms) is provided by clients. the jargon only includes specific words and phrases without the availability of speech data and domain-specific text. it challenges pre-trained asr models, e.g., whisper, and cur-rent methods of contextual speech recognition that usually require both speech and text data. to address the problem, we introduce a new method that injects domain-specific knowledge in the form of jargon into the decoding phase of asr engines. to do that, the method uses whisper as the backbone and jargon represented as a trie tree as the domainspecific context. by utilizing it as a form of the tree to manipulate the beam search decoding process and a prompt to give instructions to whisper, the method improves the performance on various datasets and the appearance of jargon in the final output. the method does not require speech data for fine-tuning asr models, so that facilitates the deployment in actual cases. in summary, the paper makes two main contributions as follows. the method is simple, effective, and easy to adapt with any encoder-decoder asr engines. the method of contextual speech recognition is perhaps the most relevant to our work @xcite . the method adjusts the output likelihoods of a neural network at each step in the beam search by a sequence probability computed from ngrams. while sharing the idea of using shallow fusion, our proposed method distinguishes two main points. first, we consider a small dictionary rather than using the text of testing data to create n-grams language models (lms) as @xcite . it makes our task to be more challenging. second, in the decoding phase, we modify the probability of a token appearing in the jargon while @xcite just simply used shallow fusion with the probability of lms. we follow the shallow fusion approach because it is simple but effective and is appropriate for asr in business cases. ## results and discussion performance comparison an important point is that the performance of initial prompting and our decoding methods is better than whisper, the strong baseline of asr on the four testing datasets. as mentioned, the initial prompt method directly embeds the jargon in the prompt that serves as the prefix when decoding audio segments. however, this method has a main limitation. the prompt is short (less than 250 tokens) and faces a challenge with a long dictionary in practical cases. in contrast, our decoding method directly adjusts the decoding process by using the trie tree representation. tokens appearing in the jargon receive higher probabilities than those that are not in the jargon. as a result, our decoding method is more general and can work with any asr engines and dictionaries. in addition, the initial prompt method does not show the efficiency on librispeech. this is because this corpus is public and already included in the pre-training of whisper, therefore, adding additional context in form of a dictionary is not necessary. the inclusion of librispeech in the pre-training of whisper also results the high score of drr on this dataset. despite the limitation, in actual cases, the initial prompt method still can be used by using matching or selection methods for dealing with a large dictionary. for example, the selection method can select relevant tokens given an input speech for recognition to reduce the number of tokens in the prompt. in this case, the combination of our decoding and initial prompt methods can retain promising results. in the case that the initial prompt method is not available, our method can still output competitive scores. for example, it is the best on librispeech and the second best of drr on hgp. it confirms the efficiency of our proposed decoding method. for the baselines, whisper (small and medium versions) obtains promising results. as mentioned, by using a huge amount of data for pre-training, whisper can work well on various domains in the multilingual setting. tcpgen does not show the efficiency on gpt synthesis and jnas datasets because these datasets have only testing sets while tcpgen requires training data for adaptation. for japanese, only hgp provides both training and testing sets. therefore, we fine-tuned tcpgen on hpc and directly used the model for gpt synthesis and jnas testing sets. the beam seach+n-grams lm method achieves competitive results even it is a quite simple method. it shows the role of beam search and language modeling for the contextual basing task of asr. however, the performance of this method is still behind our methods. among the four datasets, the improvement on the two in-house datasets is larger than jnas and librispeech, two public corpora. there are two possible reasons. first, the two in-house datasets contain more domain-specific knowledge and terms that may not appear in the training data of whisper. this, therefore, challenges the transcription of whisper in domain-specific data. second, jnas and librispeech are benchmark datasets that are easy to collect and use as the training data of whisper. it explains the reason why whisper operates well on benchmark asr corpora. the proposed method is simple and effective. it does not increase much the running time due to the efficient searching method on a small dictionary using a trie tree. human vs. automatic dictionary creation dictionaries created by humans are good at biasing, however, it's creation is costly and requires domain knowledge. we, therefore, investigated the behavior of injection methods using dictionaries created by humans or automation. as mentioned in section 3.3, the dictionaries used in the experiments were created by humans. to assess the setting of using automatic dictionary creation, we used a method as follows. first, the speech data of testing sets of the datasets were transcribed by using whisper. the transcription was then aligned with the gold texts to obtain wrong words based on word segmentation (we used mecab 4 for japanese). finally, common words were removed by using japanese or english dictionaries. note that we could not create the dictionary by humans for librispeech due to the large number of testing samples. therefore, we did not report the observation on librispeech. ## conclusion this paper introduces a new method for improving the performance of asr engines, i.e., whisper by taking into account jargon. to do that, the method considers the jargon as the context injected into the decoding process. domain-specific tokens receive more attention by adjusting the score of tokens in the beam search. experimental results on three japanese and one english datasets confirm two important points. first, the jargon can provide useful domain knowledge to improve the quality of whisper. it shows that the improvement on domain-specific corpora is higher than public datasets due to the lack of domain knowledge of whisper. second, the combination of our decoding and the initial prompt methods achieves the best results. the proposed method provides a simple but effective way for domain adaptation of whisper without accessing speech data for fine-tuning asr models. future work will confirm the efficiency of the method on other datasets and improve the decoding process using graph neural networks.