MTG : A Benchmark Suite for Multilingual Text Generation (2022)

Abstract: We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation. It is the first-proposed multilingual multiway text generation dataset with the largest human-annotated data (400k). It includes four generation tasks (story generation, question generation, title generation and text summarization) across five languages (English, German, French, Spanish and Chinese). The multiway setup enables testing knowledge transfer capabilities for a model across languages and tasks. Using MTG, we train and analyze several popular multilingual generation models from different aspects. Our benchmark suite fosters model performance enhancement with more human-annotated parallel data. It provides comprehensive evaluations with diverse generation scenarios. Code and data are available at https://github.com/zide05/MTG .

## introduction natural language generation (nlg) aims to automatically generate meaningful texts with the input in different formats, such as images @xcite , tables @xcite or texts @xcite . the generated texts generally target at realizing an underlying communicative goal while remaining coherent with the input information and keeping grammatically correct. multilingual text generation extends the natural language generation task to produce texts in multiple languages, which is important to overcome language barriers and enable universal information access for the world's citizens @xcite @xcite . to achieve this goal, various multilingual text generation datasets have been proposed. some of them do not incorporate cross-lingual pairs @xcite . this limits the knowledge transfer from one language to another. others involve cross-lingual pairs while english is included on either source or target side in most cases @xcite , leading to difficult transfer between low-resource or distant language pairs. constructing a multilingual text generation dataset that can directly transfer knowledge between any two languages is still under-explored. to this end, we propose mtg, a humanannotated multilingual multiway dataset. multiway means that the same sample is expressed in multiple languages. it covers four generation tasks (story generation, question generation, title generation and text summarization) across five languages (english, german, french, spanish and chinese). we do not include multilingual machine translation because mt itself is a standard task. the multiway parallel feature enables cross-lingual data construction between arbitrary language pairs. such direct parallel signal promotes knowledge transfer and cross-lingual generation between any language pairs (even distant pairs such as spanish-chinese) without involving an intermediate language such as english @xcite . the multilingual multiway feature also enables various training and test scenarios. in this paper, we design four scenarios to verify the advantages of our mtg from different aspects. several representative pretrained multilingual models are employed to test these scenarios, including multilingual bert (m-bert) @xcite , xlm @xcite , mbart @xcite and mt5 @xcite . we leverage various metrics to assess the coherence and diversity of the outputs generated by these models. besides, we also propose an ensemble metric, which mainly focuses on relevance, measuring to what degree is the generated text close to humanlevel. human evaluation is also conducted to validate models' performances. in summary, the contributions of this paper are listed as follows: (i) we propose a new human-annotated multilingual multiway text generation benchmark suite mtg. (ii) we design a new evaluation metric measuring how a text resembles human writing and prove that it has higher correlation scores with human scores compared with other automatic relevance metrics. (iii) we evaluate several representative pretrained multilingual models on our proposed mtg and make a rigorous analysis to verify its advantages. ## related work a significant body of works have been committed to the construction of multilingual datasets covering diverse tasks @xcite @xcite . xtreme @xcite ) is a multilingual understanding benchmark across 40 languages and 9 tasks, but it does not cover any generation task. @xcite propose x-factr, which is a cross-lingual factual retrieval benchmark. @xcite propose mkqa, an opendomain question answering evaluation dataset covering 26 diverse languages. @xcite present wikilingua, which is a large-scale, multilingual dataset for cross-lingual abstractive summarization systems. mlsum @xcite is a dataset for text summarization in 12 languages. wiki-40b @xcite is a multilingual language model dataset across 40+ languages. although these datasets cover multiple languages, they either belong to natural language understanding tasks or a single, specific generation task, which limits researchers to obtain general findings incorporating a set of generation tasks. xglue @xcite is a cross-lingual benchmark dataset for nine understanding tasks and two generation tasks. gem @xcite is a newly-presented vision-language dataset covering 11 image-language and video-language tasks and 32 languages. these two datasets encompass multiple tasks and languages. however, a remarkable difference of our mtg from xglue and gem is that mtg focuses on text-to-text generation tasks and is parallel across all languages, which facilitates easier knowledge transfer. ## dataset collection and methodology this section will introduce how to create the benchmark suite for multilingual text generation (mtg). in order to construct multiway parallel dataset, the initial dataset is translated into other languages by an off-the-shelf translation model. part of the translated data is randomly selected for further human annotation to increase data quality. the selection of tasks, initial datasets and languages are based on several principles as described below. ## experiments in this section, we conduct extensive experiments to benchmark the difficulty of our proposed mtg via several state-of-the-art multilingual models under different scenarios. ## leaderboard we build a leaderboard for mtg foot_5 . it provides an overall evaluation of models in two scores: mtgscore mtgscore is designed to evaluate the multilingual model. it is the average of ensemble scores over all languages and tasks. mtgscore-xl mtgscore-xl is a special score for mtg. it enbales better evaluation of crosslingual generation ability by testing model in 25 cross-lingual directions. it is the average of ensemble scores over all tasks and all cross-lingual language directions. the mtgscore and mtgscore-xl for the four multilingual pretrained models are shown in ## discussions considering the annotation cost, it is not realistic to construct a multiway text generation dataset with all data annotated by human. as a consequence, most of the non-english data in mtg are automatically translated from their english counterparts. although the n-gram consistency check when round-trip translating the data can guarantee the quality of them to some extent, some translation errors are inevitable. mtg with more annotated data and with data filtered by more reliable methods will be explored in the future. on the other hand, human often gives an overall evaluation of a generated text rather than measuring it in fine-grained aspects of grammar, fluency and relevance. thus we try to propose a metric measuring how a text resembles human writing and consider grammar, fluency and relevance as a whole. this metric may not be perfect, but it is a promising direction as there does not exist a really reliable text generation metric nowadays. ## conclusion in this paper, we propose a multilingual multiway benchmark mtg for text generation. it contains four typical generation tasks: story, question, title generation and text summarization. the key feature of mtg is that it has multiway parallel data across five diverse languages: english, german, french, spanish and chinese. it provides the benchmark with the ability to create cross-lingual data between any two languages and makes the semantic fusion between languages easier. on the other hand, it provides more evaluation scenarios, such as multilingual training, cross-lingual generation and zeroshot transfer. we also benchmark state-of-the-art multilingual pretrained models on our mtg from different metrics (including a newly proposed ensemble metric) to explore its features and promote research in multilingual text generation.