MOSPC : MOS Prediction Based on Pairwise Comparison (2023)

Abstract: As a subjective metric to evaluate the quality of synthesized speech, Mean opinion score(MOS) usually requires multiple annotators to score the same speech. Such an annotation approach requires a lot of manpower and is also time-consuming. MOS prediction model for automatic evaluation can significantly reduce labor cost. In previous works, it is difficult to accurately rank the quality of speech when the MOS scores are close. However, in practical applications, it is more important to correctly rank the quality of synthesis systems or sentences than simply predicting MOS scores. Meanwhile, as each annotator scores multiple audios during annotation, the score is probably a relative value based on the first or the first few speech scores given by the annotator. Motivated by the above two points, we propose a general framework for MOS prediction based on pair comparison (MOSPC), and we utilize C-Mixup algorithm to enhance the generalization performance of MOSPC.The experiments on BVCC and VCC2018 show that our framework outperforms the baselines on most of the correlation coefficient metrics, especially on the metric KTAU related to quality ranking. And our framework also surpasses the strong baseline in ranking accuracy on each fine-grained segment. These results indicate that our framework contributes to improving the ranking accuracy of speech quality.

## introduction speech quality evaluation metrics are designed to reflect the speech quality of synthesized speech. speech quality evaluation metrics include objective metrics @xcite @xcite and subjective metrics @xcite . mos prediction is the task of constructing an automatic evaluation metric by fitting the subjective evaluation metric mos. the training process of previous works mainly focus on predicting the mos of a single speech. by reviewing the anno-tation process of mos, we found that comparison may be a potential scoring strategy employed by some of the annotators. specifically, in the dataset vcc2018, each annotator scored an average of 226 speech. as each annotator annotates multiple speech in succession, the scores given by some annotators may be relative scores after comparison (e.g., the first or first few utterances scored by the annotator may be used as a benchmark). moreover, compared with predicting the specific mos score values of the speech samples, ranking the quality of speech samples has more practical application value and is often more difficult when speech samples have close mos scores. many previous works @xcite have raised the problem of generalization, and the performance will be significantly degraded when facing the outof-distribution (ood) problems. motivated by the above points, we propose a mos prediction model based on pairwise comparison (mospc). our contributions can be summarized as follows: to verify that mospc contributes to speech quality ranking, we test the ranking accuracy on the validation set on fine-grained mos score segments. then we utilize the c-mixup algorithm to enhance the performance of generalization on bvcc. ## related work a classic work in mos prediction task is mosnet @xcite , which adopts the model structure of cnn-bilstm and proposes a loss function combining frame-level loss and utterance-level loss. due to the need for manual annotation, few data can be used in the mos prediction task. to reduce data waste, mbnet @xcite proposes a mos predictor consisting of a meannet and a biasnet. ldnet @xcite observed that mbnet removes biasnet at inference and only retains meannet, which is inefficient. therefore, ldnet improves mbnet by adopting an encoder-decoder structure to reduce the waste of parameters. ddos @xcite proposes to eliminate the domain mismatch between self-supervised learning (ssl) model and mos prediction data, and adds score distribution of each utterance to model learning. utmos @xcite is based on ensemble learning of strong and weak learners. fusion-ssl @xcite uses late fusion, and fuses the results of 7 ssl models to predict mos value. @xcite makes a analysis of the ood problem of mos prediction. the ood problems in mos prediction mainly include unseen-system, unseen-listener, unseen-speaker in the test and validation sets. our proposed mospc adopts dynamic pairwise comparison. compared with the previous methods @xcite @xcite , our method pays more attention to correctly evaluating the relative quality of speech. ## method in this section, we will introduce the overall structure of mospc and the implementation of pairwise comparison, as well as the c-mixup algorithm used to enhance generalization performance. ## experiments and discussion in this section, we will compare the performance of mospc with the baselines @xcite @xcite and strong baseline fusion-ssl @xcite on the datasets bvcc and vcc2018. we test the generalization performance on bc2019 and vcc2016. we also list the ranking accuracy of fine-grained mos segments. ## conclusion this paper proposes a general framework for mos prediction based on pairwise comparisons (mo-spc) to solve the problem that it is difficult for mos prediction models to correctly rank speech quality when the mos scores are close. the main track experiment results show that mospc outperforms baselines on most of the correlation coefficient metrics, especially on the metric ktau related to speech quality ranking. moreover, mospc surpasses the strong baseline in ranking accuracy on each fine-grained segment. these results indicate that training in a pair comparison manner contributes to improving ranking accuracy. we leverage c-mixup algorithm to enhance the generalization performance. on the ood datasets vcc2016 and bc2019, our method outperforms baselines on all metrics. we also analyze the performance on fine-grained ood categories. our method performs better for the unseen-listener ood category than for the unseen-speaker and unseen-system ood categories.