S tream V oice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024)

Abstract: Recent language model (LM) advancements have showcased impressive zero-shot voice conversion (VC) performance. However, existing LM-based VC models usually apply offline conversion from source semantics to acoustic features, demanding the complete source speech and limiting their deployment to real-time applications. In this paper, we introduce StreamVoice, a novel streaming LM-based model for zero-shot VC, facilitating real-time conversion given arbitrary speaker prompts and source speech. Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporal-independent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech. To address the potential performance degradation from the incomplete context in streaming processing, we enhance the context-awareness of the LM through two strategies: 1) teacher-guided context foresight, using a teacher model to summarize the present and future semantic context during training to guide the model’s forecasting for missing context; 2) semantic masking strategy, promoting acoustic prediction from preceding corrupted semantic and acoustic input, enhancing context-learning ability. Notably, StreamVoice is the first LM-based streaming zero-shot VC model without any future look-ahead. Experiments demonstrate StreamVoice’s streaming conversion capability while achieving zero-shot performance comparable to non-streaming VC systems.

## introduction voice conversion (vc) aims to transfer a speaker's voice to that of another speaker without changing the linguistic content. this technique has been deployed in many real-world applications, such as movie dubbing, privacy protection, pronunciation correction, etc. with the help of neural semantic features, such as the bottleneck feature (bnf) from an automatic speech recognition (asr) system, converting source speech from arbitrary speakers in the wild has been successfully achieved @xcite . meanwhile, converting to an arbitrary target speaker with only one utterance of this speaker, called zero-shot vc, has also been researched recently @xcite . however, most existing zero-shot vc models are designed for offline systems, which are insufficient to meet the recent growing demands of streaming capability in real-time vc applications, such as live broadcasting and real-time communication @xcite . in this study, we focus on the streaming zero-shot vc as illuminated in fig. 1 . disentangling speech into different components, e.g., semantic content and speaker timbre, plays an important role in the zero-shot vc task @xcite @xcite . recently, benefiting from the powerful lm framework and the scaling up of training data, lm-based vc models @xcite @xcite with built-in in-context learning ability can learn the context relations between source and target speaker's utterances to capture fine-grained speaker timbre, achieving impressive zero-shot vc performance. however, demanding the complete source speech utterance limits these lm-based vc models to realtime scenarios; thus, they can only be used in offline applications. while several non-lm-based methods @xcite have been proposed for streaming zero-shot vc, the performance fails to generalize well to unseen speakers with high speaker similarity and speech naturalness, mainly due to the limited model capacity to scale up training data, and also the performance degradation caused by the missing future information in streaming scenario. inspired by the success of lm-based models in zero-shot vc, we aim to explore the feasibility of lms for the streaming vc scenario. an intuitive way is to follow the popular recognize-synthesis framework shown in fig. 1 , in which speech is represented in semantic bnf and acoustic features respectively extracted by a streaming asr and an audio codec. then, the lm-based vc model undertakes the transformation of semantic information into acoustic features with the target speaker's timbre. however, the development of the lm-based model in streaming zero-shot vc is hampered by two primary challenges. in this work, we propose streamvoice, a streaming lm-based model for high-quality zero-shot vc. specifically, streamvoice has a streamable architecture that integrates a single-stage language model that casually generates acoustic codecs with the collaboration of an acoustic predictor. alternating input of semantic and acoustic features at each time step ensures seamless streaming behavior. two methods are introduced to enhance the context-awareness of the lm to mitigate the performance gap caused by missing contextual information. 1) we incorporate a teacher-guided context foresight, where the vc model is taught by a teacher non-streaming asr to infer the present and future semantic information summarized by the teacher, which is then used to enhance the acoustic prediction. 2) to enhance the context learning from the input history, semantic masking encourages acoustic prediction from the preceding acoustic and corrupted semantic input, which also implicitly creates an information bottleneck to reduce the source speaker's information. experiments demonstrate streamvoice's ability to convert speech in a streaming manner with high speaker similarity for both seen and unseen speakers while maintaining performance comparable to non-streaming vc systems. as the first lm-based zero-shot vc model without any future look-ahead, the total pipeline only has 124 ms latency to perform the conversion, 2.4x faster than real-time on a single a100 gpu without engineering optimizations. converted samples can be found in https: //kerwinchao.github.io/streamvoice/ . ## conclusions this paper introduces streamvoice, a novel lmbased zero-shot vc system designed for streaming scenarios. specifically, streamvoice employs a single-stage framework encompassing a contextaware lm and an acoustic predictor. the casual design of the model's input and structure ensures compliance with streaming behavior. to address performance degradation caused by missing complete contextual information in streaming scenarios, context-aware lm adopts teacher-guided context foresight to make the model able to forecast the current and future information given by a teacher. besides, semantic masking is introduced in lm to enhance context learning from historical input and facilitate better disentanglement. finally, an acoustic predictor collaborates with the lm to generate the target speech. experiments demonstrate that streamvoice achieves streaming zero-shot vc while maintaining performance comparable to nonstreaming vc systems.