AMPS : ASR with Multimodal Paraphrase Supervision (2025)

Abstract: Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition (ASR) systems. In this work, we present a new technique AMPS, that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages, including Hindi, Marathi, Malayalam, Kannada, and Nyanja. We use paraphrases of the reference transcriptions as additional supervision while training the multimodal ASR model and selectively invoke this paraphrase objective for utterances with poor ASR performance. Using AMPS with a state-of-the-art multimodal model SeamlessM4T, we obtain significant relative reductions in word error rates (WERs) of up to 5%. We present detailed analyses of our system using both objective and human evaluation metrics.

## introduction automatic speech recognition (asr) systems have shown considerable progress in recent years but still falter when subjected to spontaneous conversational speech containing disfluencies, loosely articulated sounds, and other noise factors @xcite . this degradation in asr performance could be largely attributed to the unavailability of labeled spontaneous speech in most languages. how can we effectively utilize the limited quantities of existing labeled spontaneous speech? towards this, we propose amps (asr with multimodal paraphrase supervision) that augments an existing multilingual multimodal asr system with paraphrase-based supervision to improve asr performance on spontaneous speech in multiple languages. unlike standalone asr models that are exclusively trained to perform asr, multimodal models (such as speecht5 @xcite , maestro @xcite , etc.) are trained on multiple * these authors contributed equally to this work. amps foot_0 leverages the multimodal nature of seam-lessm4t by introducing a paraphrasing objective jointly with asr. along with using spontaneous speech and its corresponding transcription to train the speech-to-text pathway in seamlessm4t, amps also uses paraphrases of the reference transcriptions as additional supervision to train the text-totext pathway. we selectively employ paraphrasebased augmentation during training when the asr loss is high (as determined by a predetermined threshold); high asr loss is typically triggered by noise or poorly enunciated words in spontaneous speech. this selective intervention offers the model an alternate path of opting for semantically close words and phrases when the audio is not very clear. it is important that the paraphrases should not significantly differ in word order from the original transcripts, thus enabling the model to easily align representations of speech, text, and its paraphrase. with amps, we derive significant improvements in asr for spontaneous speech in hindi, marathi, malayalam, kannada, and nyanja compared to strong asr-only finetuned baselines. we report improvements not only in terms of word error rate (wer) reductions but also using semantic evaluation metrics. we also conduct a detailed human evaluation comparing the outputs of amps with the outputs from finetuning only with the asr objective and show consistent improvements in human scores. we also present many ablations, including different paraphrasing techniques, the influence of varying thresholds on the performance of amps, and using varying amounts of training data. we envision that techniques like amps could be used to improve asr of atypical speech for people with speech impairments where comprehensibility of the transcripts is critical (more than faithfulness of transcripts to the underlying speech, as highlighted in very recent work by @xcite ). ## related work in recent years, multimodal models for speech recognition have gained significant recognition @xcite @xcite . these models are capable of processing both speech and text inputs and can be adapted for tasks such as translation and speech generation. a notable example is meta ai's seamlessm4t @xcite , which can support nearly 100 languages. one of the key advantages of such models is their ability to exploit text-only training to fine-tune shared parameters in the asr pipeline. some of the recent work on text-based adaptation for asr models include for leveraging text-only data for asr finetuning is through training the text decoder with a paraphrasing objective. emerging research @xcite has shown that text paraphrasing can be used to augment llm performance but we are the first to show how paraphrases can be used to improve asr. @xcite is a recent study focusing on meaning preservation in disordered speech transcription, but do not offer any technique to help improve meaning preservation in asr outputs. ## methodology amps scaffolds on a multimodal base model comprising a speech encoder, a text encoder, and a shared decoder that takes inputs from both encoders. seamlessm4t is an example of such a model, capable of performing multiple tasks including text-to-text translation (t2t), and speech-totext transcription/translation (s2t). we introduce a new auxiliary task of text-to-text paraphrasing. this allows the model to predict words that are semantically similar and fit within the context of the sentence, without significantly altering its word order. the shared decoder architecture of seam-lessm4t allows us to exploit common parameters of both s2t and t2t pipelines and enhance the asr performance of the model. text @xmath0, consider a speech utterance given a labeled instance {x, y, y ′ }, the asr, paraphrase, and the amps loss functions are as follows. for each batch, we pass the audio through the s2t pathway and compute the asr loss between the predicted and ground-truth transcriptions. we also pass the ground-truth transcriptions as input through the t2t pathway with paraphrase-based supervision to compute l par . figure 1 illustrates a schematic of our proposed architecture. amps τ : loss function thresholding. we aim at improving the model's performance in noisy regions where the asr loss is high by selectively triggering the paraphrase objective only when the asr loss exceeds a predefined threshold τ . thus, the loss for the system is given by where τ is a hyperparameter chosen based on asr validation losses. henceforth, amps with the best threshold will be referred to as amps τ . τ values for various experiments are in appendix a. ## experimental setup for all our experiments, we use the seamlessm4t multilingual multimodal model @xcite . the text encoder and decoder modules are initialized using meta's no language left behind (nllb) model @xcite . the speech encoder in seamlessm4t uses wav2vec-bert 2.0 @xcite , which is trained on over a million hours of unlabeled speech data. further model details are in appendix b.1. datasets. the indicvoices dataset @xcite is a large collection of natural speech (74% extempore, 17% conversational and 9% read) in 22 indic languages. among the languages we chose, marathi, kannada, and malayalam are classified as low-resource by seamlessm4t (communication et al., 2023), while hindi is medium-resource. indicvoices is the only multilingual open-source indian speech corpus containing spontaneous speech and amongst the very few sources published after seamlessm4t's release. 2 we also performed experiments on nyanja (a low-resource language from zambia) from the zambezi-voice dataset @xcite . we use roughly 50 hours of (predominantly conversational, henceforth referred to as mixed) training data for each of the four indian languages. for 2 this dataset was chosen also to ensure that there was no data leakage between the seamlessm4t training data and the evaluation sets. hindi, we also simulate a very low-resource setting with random 5-hour samples of mixed and read training speech. for nyanja, we used 5 hours of training data. (for indic languages, our test sets are the validation sets that are part of indicvoices. for nyanja, we use the existing test set.) given the limited amount of training data, we use parameterefficient finetuning of adapter layers @xcite in the speech encoder and text decoder layers of the seamlessm4t model; more implementation details are in appendix b.2. paraphrasing. we translated the reference transcriptions into english using indictrans-2 @xcite for the indic languages and nllb @xcite for nyanja before translating them back to their original languages. for the hindi mixed 5-hr setting, we experimented with top-k, @xmath1, and nucleus (top-p , @xmath2.95) sampling during round-trip translation to produce more diverse paraphrases. we also explored generating paraphrases using the multilingual llm aya-23 @xcite . the exact prompt and other details are in appendix c and d.2. we used round-trip translation-based paraphrases for all the 50-hour experiments due to poor-quality llm paraphrases for low-resource languages like malayalam. evaluation metrics. evaluation metrics used were word error rate (wer), meteor and the f1 score provided by bertscore. more details are provided in appendix e. ## experiments and results improvements from asr to amps τ for these hardest 100 predictions are labeled ∆hard in we see that ∆hard consistently exceeds ∆all, indicating that the most improvement is observed in cases where pure asr performs poorly. this supports the thresholding approach that triggers the paraphrase loss only when pure asr predictions fall below a threshold. from our manual inspection of hindi samples in the hardest-100 subset, we observe examples where pure asr tends to produce acoustically similar but incorrect words, while amps τ correctly identifies the words. for example, pure asr misrecognized "hua" (meaning 'is') as "ugwa" (meaning 'grows') in a hindi example; amps τ gets this example right. ## acknowledgements the authors thank the anonymous reviewers for their constructive feedback that improved the quality of the draft. the last author gratefully acknowledges support from the amazon iitb ai ml initiative.