Beyond Common Words: Enhancing ASR Cross-Lingual Proper Noun Recognition Using Large Language Models (2024)

Abstract: In this work, we address the challenge of cross-lingual proper noun recognition in automatic speech recognition (ASR), where proper nouns in an utterance may originate from a language different from the language in which the ASR system is trained. We enhance the performance of end-to-end ASR systems by instructing a large language model (LLM) to correct the ASR model’s predictions. The LLM’s context is augmented with a dictionary of cross-lingual words that are phonetically and graphemically similar to the potentially incorrect proper nouns in the ASR predictions. Our dictionary-based method DiP-ASR (Dictionary-based Prompting for Automatic Speech Recognition) significantly reduces word error rates compared to both the end-to-end ASR baseline and instruction-based prompting of the LLM without the dictionary across cross-lingual proper noun recognition tasks involving three secondary languages.

## introduction in recent years, automatic speech recognition (asr) systems have advanced significantly for resource-rich languages, largely due to the availability of extensive labeled speech datasets @xcite @xcite . however, such asr systems often still struggle with words which are rare in the training data or missing from it @xcite . proper nouns, in particular, pose these challenges, leading to frequent misinterpretations by asr systems @xcite . these issues are further exacerbated in a cross-lingual setting -in which at inference time most of the words of the spoken sentences are from the language on which the asr system is trained, but the proper nouns may be from a secondary language, leading to a large number of out of vocabulary words. ## methodology prompting llms for cross-lingual proper noun recognition assumes that the llm's pre-training dataset includes a substantial number of crosslingual proper nouns. however, this is not the case for all languages, such as some indian languages. llms are predominantly pre-trained on datasets that contain mainly english proper nouns with limited representation of cross-lingual proper nouns. as a result, llms struggle to accurately recognize and correct cross-lingual proper nouns. to address this challenge, we introduce two methods, instruction-based prompting and prompting with dictionary. prompt template ### system: you are an automatic speech recognition error correction tool ### user: perform error correction on the top 5 output generated by an automatic speech recognition (asr) system. the asr hypothese are : hypothesis 1: 'launch ward dumda song unsportified' hypothesis 2: 'launch word dumka song unsportified' hypothesis 3: 'launch wad dumda song unsportified' hypothesis 4: 'launch wat dumda song unsportified' hypothesis 5: 'launch wat drum song unsportified' dictionay: 'unsportified' can be @xcite , 'ward' can be @xcite , 'dumda' can be @xcite , 'wad' can be @xcite ... please provide only one corrected asr hypothesis for the given utterance (hypothesis) using dictionary, do not add any extra word or explanations. ground truth: launch what jhumka song on spotify 0-shot: launch ward dumda song unsportified 1-shot: launch word dumka song unsportified few-shot (tf): launch ward jhumka song on spotify few-shot (n-tf): launch word jhumka song on spotify ## experimental setup for our experiments, we utilized the pre-trained wav2vec2-large-xlsr-53 model @xcite from huggingface, fine-tuning it with the english commonvoice corpus v6.1 for both training and validation. during decoding, the e2e-asr model employed a beam size of 5, generating a 5-best list of hypotheses for each utterance, referred to as the 'n-best' hypotheses. minimal postprocessing was applied, with each hypothesis converted to lowercase without further modifications. the effectiveness of these n-best hypotheses is detailed in the section 4. ## results and analysis our experiments demonstrate that the few-shot learning approach significantly outperforms the zero-shot and one-shot methods shown in in ## conclusion we propose an instruction-based prompting method that enhances the llm's ability to correct proper nouns, with effectiveness varying based on the pre-trained dataset of the llms. additionally, we introduce a novel approach for incorporating cross-lingual proper noun knowledge into llms. our method, dip-asr, shows significant improvements in cross-lingual proper noun recognition. our analyses of dip-asr highlight its benefits and limitations under various conditions.