To achieve this, we employ a conditional flow matching (CFM) model to sample the Mel spectrogram, given speech tokens, reference speech and speaker embedding as conditions. In the CFM model, the distribution of target Mel spectrogram is described by a probability density path from a prior distribution and the data distribution. For sampling efficiency, we employ the optimal-transport (OT) flow to match the vector field, which is given by an ordinary differential equation (ODE). A causal convolutional Transformer UNet is employed to learn the ODE with the up-sampled token, masked Mel spectrogram, speaker embedding and timestep as the conditions. At the training stage, the masked Mel spectrogram is obtained by randomly masking out 70% to 100% of the final frames in the target Mel spectrogram. As for the inference, it is provided by the Mel spectrogram extracted from the reference speech. By minimizing the L1 loss between the predicted and ground-truth ODE, we can optimize the UNet parameters. At the training stage, the timestep follows a uniform distribution. However, during the inference, we employ the cosine scheduler to offer more steps for the initial generation stage. Besides, we also train the model on both conditional and non-conditional situations to enable the classifier-free guidance (CFG) at the inference stage. The CFG strength and the number of flow estimation (NFE) are set to 0.7 and 10, respectively, according to the experimental results. The current flow matching models always work on a offline mode, i.e., only all the speech tokens are generated, the Mel spectragram can be sampled, which is not friendly for the streaming synthesis. To overcome this issue, we treat the multi-step flow estimation as a stacked deeper neural network, which repeats the UNet ten times. Thus, by making the unfolded neural network causal, we can apply it on the streaming synthesis. We construct four masks to satisfy different application situations: Non-causal Mask, Full-causal Mask, Chunk-M Mask, and Chunk-2M Mask. For each training case in a mini-batch, we randomly sample a mask from the above four masks under the uniform distribution. In this manner, one flow matching model can be compatible to different scenarios, lowering the deployment complexity. Another advantage of this chunk-aware training is that the masks with more context sever as a teacher for the ones with less context, benefiting from the implicit self-distillation scheme. Latency Analysis for Streaming Mode The first-package latency is an important metric for streaming synthesis models, which significantly affects the user experience especially in LLM-based voice chat applications. In the context of TTS, the to-synthesize text is known in advance, and the latency comes from the aspects of speech token generation, Mel spectrogram reconstruction and waveform synthesis. Thus, the first-package latency of CosyVoice 2 can be obtained by summing three components: M times the computation time of the LM to generate one speech token, M times the computation time of the Flow Matching model to generate the Mel spectrogram frames for one speech token, and M times the computation time of the vocoder to synthesize waveforms for one speech token. In the context of LLM-based voice chat, the length of first-package-required text should also be considered. The first-package latency becomes less than or equal to the sum of N times the computation time of a LLM to generate one text token, plus the TTS latency. Note that, since the multi-character tokens are masked out in CosyVoice 2's text tokenizer, the text tokens used by text LLMs always encode longer raw text than those of CosyVoice 2. Thus, the the first-package latency must be lower than the calculated sum. Instructed Generation  To enhance the controllability of CosyVoice 2, we integrated the instructed dataset into the base training set. We have collected 1500 hours of instructed training data, which includes both natural language instructions and fine-grained instructions. For natural language instructions, we prepend a natural language description and a special end token, ``
'' before the to-synthesize input text. These descriptions cover aspects such as emotion, speaking rate, role-playing, and dialects. Emotions include Happy, Sad, Surprised, Angry, Fearful, Disgusted, Calm, and Serious. Speaking rates include Fast, Very Fast, Slow, and Very Slow. Dialects include Cantonese, Sichuan, Shanghai, Zhengzhou, Changsha, and Tianjin. Role-playing examples include Mysterious, Fierce, Curious, Elegant, Lonely, Robot, and Peppa Pig. For fine-grained instructions, we insert vocal bursts between text tokens, using markers like ``[laughter]'' and ``[breath]''. Additionally, we apply vocal feature tags to phrases; for instance, ``<strong>XXX</strong>'' indicates emphasis on certain words, while ``<laughter>XXX</laughter>'' signifies speaking with laughter. Multi-Speaker Fine-tuning Fine-tuning the pre-trained model on specific speakers (SFT) can further improve the generation quality and speaker similarity. In this report, we introduce the multi-speaker fine-tuning (mSFT), in which the pretrained model is fine-tuned on multiply speakers simultaneously rather than a single speaker. This approach ensures comprehensive prosody and pronunciation coverage across multiple speakers and mitigates potential catastrophic forgetting from the pretrained models. To avoid timbre confusion between various speakers, we prepend speaker-prompt tags, like ``Speaker A
'' to the input text for a specific speaker. If a training sample is not labeled to a speaker, a special tag, ``unknown
'', is utlized. The learning rate is set to 1e-5 during the whole multi-speaker fine-tuning process. Reinforcement Learning for SFT Reinforcement learning is a commonly used method in the training of large language models, which can make the LM output align with human preference. In CosyVoice 2, we employ speaker similarity (SS) and recognition word error rate (WER) from the ASR system as the reward function to improve speaker similarity and pronunciation accuracy in the fine-tuning stage. We use WER and SS to distinguish preferred samples and rejected samples and optimize the TTS system with direct preference optimization (DPO). However, this method is time-consuming and computation-consuming as it should synthesis the audios through the TTS system repeatedly to obtain distinguishable preference and rejected samples. During training, four forward operations are needed for one training step. To simplify the process, we recover the LM predicted token into quantized low-rank representations, and directly use the ASR backend of the speech tokenizer to re-predict the input text. Then the predicted log posterior can be regarded as the ASR reward function to optimize the text-speech language model. During training, the ASR backend parameters are frozen. As the sample operation of the tokens still prevent us to optimize the model directly, we use the gumbel softmax sampling to make it differentiated and then optimize the language model by the ASR loss. Experimental Settings  Training Data for Speech Tokenizer  A 200,000-hour dataset is used to train the speech tokenizer with normalized transcriptions as labels. The training data comes from three different resources: open source ASR datasets, internal industrial datasets and TTS generation datasets. The dataset includes 110,884 hours of Chinese and 99,918 hours of English data. Although we only used Chinese and English data when training the speech tokenizer, subsequent experiments revealed that the speech tokenizer had zero-shot capability for other languages. It can be also used for speech synthesis in languages such as Japanese and Korean. Training Data for CosyVoice 2 CosyVoice 2 shares the same training data as its previous version. We first collect the speech-only data with internal speech processing tools. Subsequently, the Paraformer and SenseVoice are employed to generate pseudo text labels for Chinese and other languages, respectively. We also employ an internal force-alignment model to filter out low-quality data and enhances the accuracy of punctuation. Data details are 130,000 hours for Chinese, 30,000 for English, 4,600 for Japanese, and 2,200 for Korean. Evaluation Settings We evaluate our CosyVoice 2 on two test sets. The first one is constructed from the test-clean set of Librispeech corpus, denoted as test-clean. This test set is used to evaluate CosyVoice 2 on a limited English domain. The Whisper-large V3 is used as the ASR model to evaluate the content consistency. As for the speaker similarity (SS), we employ the ERes2Net model to extract speaker embeddings of prompt and generated utterances, and their raw cosine similarity is treated as the speaker similarity. NMOS score is used to evaluate the objective quality. The second evaluation is conducted on the SEED test sets, which is widely used to evaluate recent TTS models, covering various text domains and reference speeches. In this evaluation, about 2,000 Chinese and 1,000 English samples are selected from CommonVoice datasets, denoted as test-zh and test-en, respectively. In addition, about 400 hard test cases are also included to evaluate the robustness of TTS models on text repetition, tongue twister and other challenging synthesis cases, denoted as test-hard in this report. The Paraformer is employed to recognize the synthesis results of test-zh and test-hard, while the Whisper-large V3 is adopted for test-en to evaluate the content consistency. We adopt two speaker verification (SV) models to evaluate speaker similarity: WavLM-finetuned SV model and ERes2Net. Benchmark for Japanese and Korean We prepare two test sets, denoted as test-ja and test-ko, for the evaluation on Japanese and Korean speech synthesis. The test-ja consists 1,000 samples extracted from the CommonVoice dataset, which are used to measure the model’s performance on various metrics, such as WER, SS, MOS. Specifically, we randomly shuffle and pair the entire CommonVoice JA-test set as reference utterance and target utterance spoken. Considering the wide range of utterances' text lengths of JA-test set, we randomly selected 1,000 pairs of reference-target utterances from the length range from 8 to 32 characters as our final test set. For the test-ko, we selected 1,000 speech samples with a WER of less than 5% and no deletion or insertion errors, utilizing the Whisper-Large V3 as the ASR model. These samples were used as reference utterances for the Korean speech synthesis. For the input text, we randomly selected 1,000 text samples from the remaining data. We have released the lists of prompt speeches, prompt transcriptions and input text from these two test sets are released to facilitate result reproduction. By providing this open-source data, we aim to establish a benchmark for evaluating Japanese and Korean TTS models. The Whisper-large V3 is used as the ASR model for Japanese and Korean evaluations. Experimental Results  Evaluations on Speech Tokenizer  An ideal speech tokenizer is supposed to effectively utilizes the codebook, preserves information at a high fidelity, and demonstrates speaker independence. In this part we evaluate our supervised speech tokenizer from four aspects: 1) Codebook utilization rate; 2) ASR error rate within the entire encoder; 3) Token visualization of different speakers; 4) Speaker identification training. It turns out that the FSQ-based tokenizer fully utilizes the codebook and maintains more effective information from the aspect of ASR, indicating more semantic information maintained by FSQ. We further analyze the characteristics of FSQ through the t-SNE visualization. As an upstream model for TTS tasks, the tokenizer should strive to minimize the entanglement of speaker identity information with the speech signal. We selected 100 speech samples from each of the three speakers in the VoxCeleb1 dataset and visualized the corresponding tokens. It is evident that before the quantization, Encoder 1's outputs exhibit different distributions among different speakers. In contrast, the distributions of quantized representations are nearly indistinguishable. In addition, the tokenizer fully utilizes the codebook. Subsequently, the S3prl toolkit is employed to further evaluate the speaker entanglement by performing speaker identification (SID) task. We use Sensevoice-large encoder with FSQ as an upstream feature extractor and train SID task with representations before or after the quantization. The convergence curves of SID training show that the SID layer with quantized tokens does not converge, which proves the decoupling function of the tokenizer on speaker information. Comparison Results with Baselines We first evaluated our CosyVoice 2 models on a limited English text domain and compared it with several open-source models, such as ChatTTS, GPT-SoVITs, OpenVoice, ParlerTTS, EmotiVoice, and its predecessor CosyVoice. The objective results include content consistency (WER), speech quality (NMOS) and speaker similarity (SS). The results show that CosyVoice 2 achieves state-of-the-art performance on the Librispeech test-clean set, surpassing all baseline models across all evaluation metrics. Notably, CosyVoice 2 even demonstrates higher content consistency, speech quality, and speaker similarity than human utterances, indicating its human-parity synthesis quality. We also evaluated CosyVoice 2 on the commonly-used test sets: SEED test-zh, test-en and test-hard, which include diverse input texts and reference speeches from various domains. On the test-zh set, CosyVoice 2 surpasses all open-sourced models in terms of CER and SS, falling short of the commercial model SEED-TTS by only a small margin. On the test-en set, CosyVoice 2 ranks fourth and third in terms of WER and SS, respectively. This may result from the imbalance in the volume of training data between Chinese and English. We plan to explore data scaling in future work to enhance content consistency in English. On the test-hard set, the offline CosyVoice 2 model achieves state-of-the-art performance across all compared baseline, demonstrating its robustness in challenging synthesis scenarios. Compared with human-generated speeches, CosyVoice 2 shows comparable content consistency and superior speaker similarity. Considering the recognition errors can also stem from the ASR model, it is reasonable to conclude that CosyVoice 2 achieves a human-parity synthesis capability. We also evaluated the streaming mode, denoted as ``CosyVoice 2-S''. For both evaluation settings, the streaming mode's performance is nearly lossless in typical test cases. Only in challenging cases is there a slight degradation in content consistency, highlighting the strength of our unified streaming/non-streaming framework. We found that the results of speaker similarity are not consistent on different SV models. This may indicate a new research topic on how to evaluate speaker similarity for TTS models automatically. Since different TTS models may use different SV models to extract speaker information, evaluating speaker similarity with the same SV model allows a more accurate evaluation on the utilization of speaker information. Therefore, we employ ERes2Net for evaluating speaker similarity in subsequent experiments. Modular Ablation Study We conducted a modular ablation study on the text-speech language model to assess the impacts of our modifications, including LLM initialization, removing speaker embedding, and utilizing FSQ.