STTATTS : Unified Speech-To-Text And Text-To-Speech Model (2024)

Abstract: Speech recognition and speech synthesis models are typically trained separately, each with its own set of learning objectives, training data, and model parameters, resulting in two distinct large networks. We propose a parameter-efficient approach to learning ASR and TTS jointly via a multi-task learning objective and shared parameters. Our evaluation demonstrates thatthe performance of our multi-task model is comparable to that of individually trained models while significantly savingcomputational and memory costs (~50% reduction in the total number of parameters required for the two tasks combined). We experiment with English as a resource-rich language, and Arabic as a relatively low-resource language due to shortage of TTS data. Our models are trained with publicly available data, and both the training code and model checkpoints are openly available for further research.

## introduction fundamentally, text and speech are different representations of similar information, with text being a far more condensed form of the linguistic content of speech. conversion between text and speech modalities in the form automatic speech recognition (asr), text-to-speech synthesis (tts), or voice conversion (vc), are traditionally achieved by training separate asr, tts, and vc models as the input and output modalities and training objectives differ significantly. however, considering the recent developments in self-supervised and multimodal pre-training, a more integrated approach can now be promising. a unified model trained simultaneously for multiple speech/text to speech/text, improves on generalization to new data, cross-task knowledge transfer, simplifies maintenance, and also reduces the computational and memory requirements for training, storage, and inference. re-cent studies seek to achieve a smooth fusion of text and audio by developing unified audio-text models capable of addressing diverse tasks both within and across these modalities. while these models are considered multi-modal if they can process different input modalities, for our purposes we group audio-text models into two categories based on their output modality: uni-modal and cross-modal. we describe uni-modal approaches as models capable of generating output in a single modality only, such as whisper @xcite , and google usm @xcite , which only generate text outputs. cross-modal approaches, on the other hand, are capable of generating outputs in both speech and text modalities, such as viola @xcite , speecht5 @xcite , and speechgpt @xcite . some of these models @xcite use discrete representations for audio tasks, merging text and audio tokens in a shared vocabulary while jointly training multiple tasks; other models, such as speecht5 @xcite use continuous representations for audio. while speecht5 is pre-trained with a cross-modal objective, handling both text and speech as input and output modalities, the model is fine-tuned individually for downstream tasks such as asr and tts. this work builds on these developments, particularly speecht5 @xcite , by achieving a truly cross-modal speech and text conversion in a single architecture. unlike previous work, our approach does not separate the speech/textto-speech/text tasks based on input/output modalities; instead, we fine-tune these components concurrently using a unified model and loss function with the help of a simple mlp-based task fusion module. the resulting model is a single encoderdecoder that can handle both modalities at the input and output, depending on the desired task. we summarize our contributions below: ## our method in this section, we describe a unified speech-to-text and text-to-speech model, sttatts, our proposed architecture for jointly training asr and tts. after unified self-supervised training as described in @xcite , we propose utilizing a multi-task loss objective to optimize our model parameters for multiple tasks, along with a task fusion module to handle the different tasks. unlike the fine-tuning methodology followed in speecht5 @xcite , which results in a completely disjoint copy of the model for each task (see figure 2 : right), sttatts utilizes a task fusion module with negligible number of additional parameters to handle multiple tasks using the same encoder-decoder backbone (see figure 1 ). this results in a ∼50% reduction in the total number of parameters required for the two tasks combined. ## experimental settings we conducted experiments across two languages: english and arabic using open-source datasets. english was used as a resource-rich language, and trained using the benchmark librispeech dataset; this enables the utilization of pre-trained models from speecht5 @xcite , and a direct comparison with their downstream models. arabic served as a relatively low-resource language, mainly because of the shortage of clean speech data that can be utilized for training high-quality tts systems @xcite . this also enables using the pre-trained checkpoints from artst (toyin et al., 2023). ## results in this section, we report our model's performance, comparison against speecht5 single-task models (see figure 2 ) and comparison against a baseline joint-task model, voxlm. ## discussion we described our experiments of jointly learning speech-to-text and text-to-speech models based on the speecht5 architecture, resulting in a truly multi-modal and functional model, that accepts both speech and text as input and output. we experimented with both arabic and english languages, with arabic being a relatively low-resource language due to limited amounts of data available for tts training. our results show that while it is possible to train models using our framework with less than 20 hours of speech in total, more data is always better for maximizing performance in both asr and tts tasks. our results are the first to report multi-task asr/tts in the arabic language, showing promising results in low-resource settings, and potential for improvement with additional data. for english, a few other models have been recently proposed; our comparative analysis with the only publicly available model of this variety, named voxtlm @xcite , favors sttatts in both performance and parameter efficiency. we experimented with different parameter-efficient ap-proaches to jointly learn asr and tts tasks and we find using the task-fusion module strikes a perfect balance in performance between both tasks with the least amount of parameters. it's also worth noting that the task-fusion module makes incorporating more tasks and output modalities feasible as the module aligns latent representation to match the desired output modality. furthermore, the integrated multi-task approach, in addition to being more efficient in model size, is more efficient in training as it requires fewer updates in total. future work will explore the possibility of integrating additional text output tasks within the framework and improving synthesized speech naturalness. we believe that the proposed model, being trained on publicly available data with the code and checkpoints publicly available foot_5 , can serve as a strong baseline for multi-task speech processing. limitations. our experiments focus on three key aspects: language generalization, scalability (in terms of training data requirements and tasks), and parameter efficiency. although we explored two languages separately, we did not experiment with a joint multilingual models. additionally, we used voxlm as our only baseline for multi-task models. while other models have recently been proposed, their code and models are not yet publicly available for comparison. we did not explore increasing model size, as it would require pre-training from scratch, which is computationally expensive. however, slightly larger models could potentially enhance performance in the multi-task setting by better embedding the diverse input and output modalities. finally, we found that subjective mos evaluation was rather difficult to conduct as most outputs were intelligible but somewhat noisy and unnatural. for arabic, the small data size and lack of diacritics does result in degraded intelligibility due to mismatched pronunciation of short vowels. therefore, the reported tts results provide some signal of quality, but may not be informative of the actual quality of the synthesized speech.