V i T - TTS : Visual Text-to-Speech with Scalable Diffusion Transformer (2023)

Abstract: Text-to-speech(TTS) has undergone remarkable improvements in performance, particularly with the advent of Denoising Diffusion Probabilistic Models (DDPMs). However, the perceived quality of audio depends not solely on its content, pitch, rhythm, and energy, but also on the physical environment.In this work, we propose ViT-TTS, the first visual TTS model with scalable diffusion transformers. ViT-TTS complement the phoneme sequence with the visual information to generate high-perceived audio, opening up new avenues for practical applications of AR and VR to allow a more immersive and realistic audio experience. To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information. Experimental results demonstrate that ViT-TTS achieves new state-of-the-art results, outperforming cascaded systems and other baselines regardless of the visibility of the scene. With low-resource data (1h, 2h, 5h), ViT-TTS achieves comparative results with rich-resource baselines.

## introduction text-to-speech (tts) @xcite aims to synthesize audios that is consistent with the reference samples in terms of semantic meaning, timbre, emotions, and melody, and has shown remarkable advancements with the advent of denoising diffusion probabilistic models (ddpms). however, the perceived audio quality is not solely determined by these aspects, as it is also influenced by the surrounding physical environment. for instance, a room with hard surfaces like concrete or glass reflects sound waves, whereas a room with soft surfaces such as carpets or curtains absorbs them. this variance can drastically impact the clarity and quality of the sound we hear. to ensure an authentic and captivating experience, it is imperative to accurately model the acoustics of a room, particularly in virtual reality (vr) and augmented reality (ar) applications. recent years have seen a surge in significant research @xcite @xcite addressing the languagevisual modeling problem. for instance, @xcite have proposed a unified video-language pretraining framework for learning robust representation, while @xcite have focused on large-scale image-text pairs pre-training via contrastive learning. visual tts open-ups numerous practical applications, including dubbing archival films, providing a more immersive and realistic experience in virtual and augmented reality, or adding appropriate sound effects to games. despite the benefits of language-visual approaches, training visual tts models typically requires a large amount of training data, while there are very few resources providing parallel textvisual-audio data due to the heavy workload. besides, creating a sound experience that matches the visual content remains challenging when developing ar/vr applications, as it is still unclear how various regions of the image contribute to reverberation and how to incorporate the visual modality as auxiliary information in tts. in this work, we formulate the task of visual tts to generate audio with reverberation effects in target scenarios given a text and environmental image, introducing vit-tts to address the issues of data scarcity and room acoustic modeling. to enhance visual-acoustic matching, we 1) propose the visual-text fusion to integrate visual and textual information, which provides fine-grained language-visual reasoning by attending to regions of the image; 2) leverage transformer architecture to promote the scalability of the diffusion model. regarding the data shortage challenge, we pre-train the encoder and decoder in a self-supervised manner, showing that large-scale pre-training reduces data requirements for training visual tts models. experiments results demonstrate that vit-tts generates speech samples with accurate reverberation effects in target scenarios, achieving new stateof-the-art results in terms of perceptual quality. in addition, we investigate the scalability of vit-tts and its performance under low-resource conditions (1h/2h/5h). the main contributions of this work are summarized as follows: ## conclusion in this paper, we proposed vit-tts, the first visual text-to-speech synthesis model that aimed to convert written text and target environmental images into audio that matches the target environment. to mitigate the data scarcity for training visual tts tasks and model visual acoustic information, we 1) introduced a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leveraged the diffusion transformer scalable in terms of parameters and capacity to improve performance. experimental results demonstrated that vit-tts achieved new state-of-the-art results and performed comparably to rich-resource baselines even with limited data. to this end, vit-tts provided a solid foundation for future visual text-to-speech studies, and we envision that our approach will have farreaching impacts on the fields of ar and vr.