CCE val: A Representative Evaluation Benchmark for the C hinese-centric Multilingual Machine Translation (2023)

Abstract: The Chinese-centric Multilingual Machine Translation (MMT) has gained more importance recently due to increasing demands from international business development and cross-cultural exchanges. However, an important factor that limits the progress of this area is the lack of highly representative and high-quality evaluation benchmarks. To fill this gap, we propose CCEval, an impartial and representative Chinese-centric MMT evaluation dataset. This benchmark dataset consists of 2500 Chinese sentences we meticulously selected and processed, and covers more diverse linguistic features as compared to other MMT evaluation benchmarks. These sentences have been translated into 11 languages of various resource levels by professional translators via a rigorously controlled process pipeline to ensure their high quality. We conduct experiments to demonstrate our sampling methodology’s effectiveness in constructing evaluation datasets strongly correlated with human evaluations. The resulting dataset enables better assessments of the Chinese-centric MMT quality. Our CCEval benchmark dataset is available at https://bright.pcl.ac.cn/en/offlineTasks.

## introduction multilingual machine translation (mmt) is a hot research area that has made big progress recently @xcite @xcite . as the language spoken by the largest population in the world, chinese has become more and more influential globally, and china has been growing its worldwide impact. for example, in 2021, china's total import and export trade amount has reached 21% of the entire global trade amount ‡ , which is a significant proportion. despite this, the chinese-centric mmt research has received less § equal contribution. * corresponding author. a critical challenge hindering the progress of the chinese-centric mmt is evaluation, mainly due to the lack of representative and humancorrelated evaluation datasets. recently, a few evaluation benchmarks for mmt have emerged, such as flores-101 @xcite and tatoeba @xcite . however, they can only be used for english-centric translation evaluations @xcite , have not undergone a rigorous human expert annotation and quality inspection process to ensure the dataset's high quality @xcite , or are only limited to specific domains @xcite . furthermore, an ideal mmt evaluation dataset shall provide system-level evaluations that correlate closely with human assessments. to achieve a high correlation, zhan et al. @xcite selected samples yielding substantial performance fluctuations of evaluated systems, while liu et al. @xcite allocated greater weights to samples with elevated translation chunk entropy. the restrictions of these works are their dependence on hypothesis text in the source sentence sampling process. to overcome previous limitations as mentioned above, we propose a highly representative evaluation benchmark dataset, named cceval, which is specifically constructed to enable evaluation of the chinese-centric mmt models/systems. each language pair in the cceval contains 2,500 chinese source sentences we carefully selected and processed using a diversity and translation difficulty based approach. utilizing human-annotated multidimensional quality metrics (mqm) @xcite corpus from recent workshop on machine translation (wmt) evaluation campaigns, we conduct experiments to demonstrate the validity of this approach in building evaluation datasets correlated better with human assessments. these well selected chinese source sentences have been professionally translated into eleven languages of various resource levels, i.e., mongolian, lao, portuguese, polish, czech, turkish, italian, russian, arabic, greek, and croatian. note that about half of them are low-resource languages. we employ a systematic annotation and translation quality inspection workflow to guarantee the dataset's high quality. the cceval dataset is accessible via bright * , a chinese-centric mmt evaluation platform. to foster relevant research using our benchmark, we validate the effectiveness of cceval by conducting evaluation tests on actual mmt baseline models. the contributions of our work are: to the best of our knowledge, this is the first evaluation benchmark for the chinese-centric mmt research community. ## dataset construction the overall construction process of our cceval dataset includes three steps. in the first stage of source text screening, we consider various linguistic features of the chinese sentences, and screen appropriate sentences from the initial text pool to satisfy the corresponding source diversity requirements. subsequently, we propose a method to calculate the translation difficulty of each sentence, and further select highly representative samples based on this method. finally, in the target sentence translation phase, we adopt a strict translation quality inspection pipeline to obtain a high-quality dataset. ## conclusion in this paper, we construct and publicly release cceval, a chinese-centric mmt evaluation benchmark dataset covering twelve languages of varying resource levels, about half of which are lowresource languages. the dataset is highly representative, and the linguistic diversity of chinese outperforms existing mmt benchmarks. we hope that the open source of this dataset will benefit the community. in the future, we will continue to expand the number of languages covered in cceval and increase the dataset's sample size. we would horizontal axis is the top percentage for the selected subset based on the sample's difficulty score ranking. vertical axis is the kendall's tau-like correlation coefficient @xcite .