Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition (2024)

Abstract: Synthetic data is widely used in speech recognition due to the availability of text-to-speech models, which facilitate adapting models to previously unseen text domains. However, existing methods suffer in performance when they fine-tune an automatic speech recognition (ASR) model on synthetic data as they suffer from the distributional shift commonly referred to as the synthetic-to-real gap. In this paper, we find that task arithmetic is effective at mitigating this gap. Our proposed method, SYN2REAL task vector, shows an average improvement of 10.03% improvement in word error rate over baselines on the SLURP dataset. Additionally, we show that an average of SYN2REAL task vectors, when we have real speeches from multiple different domains, can further adapt the original ASR model to perform better on the target text domain.

## introduction existing automatic speech recognition (asr) models have been found to lack generalizability towards domains unseen during training @xcite @xcite . existing works, when adapting an asr model to a previously unseen domain, often rely on synthetic speech data @xcite @xcite @xcite due to its ease of generation and availability. however, this approach often leads to performance degradation due to acoustic mismatches such as intonations, background noise, speaker accents, and environmental sound differences between synthetic and real speech @xcite . this distributional shift is often referred to as the synthetic-to-real gap. this paper tackles this problem, particularly when adapting an asr model from a source domain with text and real speech data to a new target domain with only text data. our idea is that paired synthetic speech and real speech data within a single domain can guide the adaptation of models trained on synthetic data to perform better on real-world data in a new domain. inspired by the new paradigm of editing pre-trained neural networks by manipulating their weights @xcite , we propose to bridge the synthetic-to-real gap using task vectors @xcite . a task vector is a representation that encodes the difference between two tasks, allowing models to perform arithmetic operations to transition from one task to another. in this paper, we show that we can apply simple arithmetic operations to bridge the synthetic-to-real gap. taking inspiration from @xcite , we propose syn2real task vector for syntheticto-real adaptation in asr. figure 1 provides an overview of the syn2real task vector approach. the top row illustrates the process of fine-tuning models on synthetic and real speech data separately and then deriving the syn2real task vector from the differences in their parameters. the bottom row demonstrates the application of this vector to a model fine-tuned on synthetic target domain data, resulting in an adapted model with improved performance by incorporating the acoustic characteristics of real speech. we conduct comprehensive experiments and ablation studies to demonstrate the effectiveness of our approach. applying the syn2real task vector results in an average improvement of 10.03% in word error rate (wer) for unseen target domains compared to the model before applying our method. cosine similarity analysis of syn2real task vectors generated by different text-to-speech (tts) models confirms that syn2real task vectors effectively capture domain-specific acoustic information. the other direction is to adapt asr models with synthetic speech. zheng et al. (2021) develop a method that provides synthetic audio for out-ofvocabulary (oov) words to boost recognition accuracy. yang et al. (2023) works on personalizing asr with synthetic speech. bataev et al. (2023) focuses on developing a mel-spectrogram generator to improve asr models. task arithmetic the concept of task vector is introduced in @xcite . task vectors are created by subtracting the weights of a fine-tuned model from those of its corresponding pre-trained model. different task vectors derived from the same pre-trained models can then be adjusted and combined through these simple arithmetic operations such as addition and subtraction to achieve multi-task learning @xcite and task forgetting @xcite . recently, task vectors have shown promise in natural language processing (nlp) @xcite @xcite . @xcite used a task vector from a negatively fine-tuned model to mitigate hallucinations. @xcite proposed combining parameter-efficient fine-tuning (peft) modules @xcite arithmetically. @xcite obtained the chat vector by subtracting the chat version of llama 2 @xcite from its pre-trained version, enhancing dialogue capabilities and safety. @xcite introduced resta, adding a safety vector to re-align safety for models finetuned on downstream tasks. the application of task vectors is relatively underexplored in asr. @xcite applied task arithmetic to asr models and introduced a "task analogy" formulation, improving performance on low-resource tasks using models trained on high-resource tasks. unlike @xcite , we focus on using task vector to mitigate the distributional shift between real and synthetic data. ## methodology in the context of automatic speech recognition (asr) domain adaptation, domain mismatch can be broadly classified into three categories: ## experimental setups we design our experiments to answer the following questions: q1: what is the efficacy of syn2real task vector?, q2: how does syn2real task vector perform across different model sizes? q3: is syn2real task vector effective on asr models other than whisper?, q4: can we form syn2real task vectors from other tts models?, q5: what is the impact of the scaling factor λ? , q6: do syn2real task vectors obtained with the same tts have similar direction? to address these questions and simulate realworld scenarios, we first create a source domain asr model by combining synthetic and real speech data from various domains. we then adapt this source domain asr model to the target domain using data synthesized by tts models. syn2real task vector is constructed by subtracting the weights of an asr model fine-tuned on synthetic data from the weights of the same asr model fine-tuned on real data, both using the same pre-trained model as the starting point. our goal is to improve the performance of an asr model on the target domain without using any real speech from the target domain. ## results & discussion here, we discuss our results in relation to the questions we set out to answer. ## syn2real task vector given domain labels in this section, we explore an alternative approach to generating syn2real task vectors, assuming we have access to domain labels for the data in the source domains. this approach, which we refer to as syn2real ensemble task vector, involves generating separate syn2real task vectors for each source domain and then combining them to enhance the adaptation of the asr model to the target domain. ## conclusion this paper introduces syn2real task vector to address mismatches between synthetic and real speech data. future work will refine this approach and extend its application to other types of tasks and data such as visual data, contributing to more reliable speech and vision recognition systems. experiments showed significant wer reductions, averaging 10.03% across 18 domains. we test various models, including whisper small, whisper base, whisper tiny, and wav2vec2-conformer, with syn2real task vector showing robust performance. we also explore syn2real ensemble approach further enhanced domain-specific adaptation but required domain labels. overall, syn2real task vector is a promising solution for improving asr models with synthetic data.