MTG: A Benchmark Suite for Multilingual Text Generation
Abstract
We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation.
It is the ﬁrst-proposed multilingual multiway text generation dataset with the largest human-annotated data (400k). It includes four generation tasks (story generation, question generation, title generation and text summarization) across ﬁve languages (English, German, French, Spanish and Chinese).
The multiway setup enables testing knowledge transfer capabilities for a model across languages and tasks.
Using MTG, we train and analyze several popular multilingual generation models from different aspects.
Our benchmark suite fosters model performance enhancement with more human-annotated parallel data. It provides comprehensive evaluations with diverse generation scenarios. Code and data are available at https://github.com/zide05/ MTG .
Introduction
Natural language generation (NLG) aims to automatically generate meaningful texts with the input in different formats, such as images (Anderson et al., 2018), tables (Ye et al., 2020) or texts (Guan et al., 2019).
The generated texts generally target at realizing an underlying communicative goal while remaining coherent with the input information and keeping grammatically correct.
Multilingual text generation extends the natural language generation task to produce texts in multiple languages, which is important to overcome language barriers and enable universal information access for the world’s citizens (Artetxe et al., 2020; Arivazhagan et al., 2019; Pan et al., 2021).
To achieve this goal, various multilingual text generation datasets have been proposed.
Some of them do not incorporate cross-lingual pairs (Liang et al., 2020; Ladhak et al., 2020).
This limits the knowledge transfer from one language to another.
Others involve cross-lingual pairs while English is included on either source or target side in most cases (Zhu et al., 2019; Ladhak et al., 2020), leading to difﬁcult transfer between low-resource or distant language pairs.
Constructing a multilingual text generation dataset that can directly transfer knowledge between any two languages is still under-explored.
To this end, we propose MTG, a humanannotated multilingual multiway dataset. Multiway means that the same sample is expressed in multiple languages.
It covers four generation tasks (story generation, question generation, title generation and text summarization) across ﬁve languages (English, German, French, Spanish and Chinese).
We do not include multilingual machine translation because MT itself is a standard task.
The multiway parallel feature enables cross-lingual data construction between arbitrary language pairs.
Such direct parallel signal promotes knowledge transfer and cross-lingual generation between any language pairs (even distant pairs such as Spanish-Chinese) without involving an intermediate language such as English (Leng et al., 2019).
The multilingual multiway feature also enables various training and test scenarios.
In this paper, we design four scenarios to verify the advantages of our MTG from different aspects.
Several representative pretrained multilingual models are employed to test these scenarios, including multilingual BERT (M-BERT)
(Devlin et al., 2019),
XLM (Lample and Conneau, 2019), mBART (Liu et al., 2020) and mT5 (Xue et al., 2020).
We leverage various metrics to assess the coherence and diversity of the outputs generated by these models.
Besides, we also propose an ensemble metric, which mainly focuses on relevance, measuring to what degree is the generated text close to humanlevel.
Human evaluation is also conducted to validate models’ performances.
In summary, the contributions of this paper are listed as follows:
(i)
We propose a new human-annotated multilingual multiway text generation benchmark suite MTG.
(ii) We design a new evaluation metric measuring how a text resembles human writing and prove that it has higher correlation scores with human scores compared with other automatic relevance metrics.
(iii)
We evaluate several representative pretrained multilingual models on our proposed MTG and make a rigorous analysis to verify its advantages.
Related Work
A signiﬁcant body of works have been committed to the construction of multilingual datasets covering diverse tasks (Hu et al., 2020; Jiang et al., 2020;
Longpre et al., 2020).
XTREME (Hu et al., 2020) is a multilingual understanding benchmark acrosslanguages and 9 tasks, but it does not cover any generation task.
Jiang et al.
propose X-FACTR, which is a cross-lingual factual retrieval benchmark.
Longpre et al.
(2020) propose MKQA, an opendomain question answering evaluation dataset covering 26 diverse languages.
Ladhak et al.
(2020) present WikiLingua, which is a large-scale, multilingual dataset for cross-lingual abstractive summarization systems.
MLSUM (Wang et al., 2021) is a dataset for text summarization in 12 languages.
Wiki- 40 B (Guo et al., 2020) is a multilingual language model dataset across 40+ languages.
Although these datasets cover multiple languages, they either belong to natural language understanding tasks or a single, speciﬁc generation task, which limits researchers to obtain general ﬁndings incorporating a set of generation tasks.
XGLUE (Liang et al., 2020) is a cross-lingual benchmark dataset for nine understanding tasks and two generation tasks.
GEM (Gehrmann et al., 2021) is a newly-presented vision-language dataset covering 11 image-language and video-language tasks and 32 languages.
These two datasets encompass multiple tasks and languages.
However, a remarkable difference of our MTG from XGLUE and GEM is that MTG focuses on text-to-text generation tasks and is parallel across all languages, which facilitates easier knowledge transfer.
Dataset Collection and Methodology
This section will introduce how to create the benchmark suite for multilingual text generation (MTG).
In order to construct multiway parallel dataset, the initial dataset is translated into other languages by an off-the-shelf translation model.
Part of the translated data is randomly selected for further human annotation to increase data quality.
The selection of tasks, initial datasets and languages are based on several principles as described below.
3. Task and Dataset Selection
It is important to select suitable tasks for our MTG benchmark to make it diverse and challenging.
Thus, we deﬁne several criteria during the task selection procedure:
Task Deﬁnition Tasks should be well-deﬁned, which means that humans can easily determine whether the generated results meet the task requirements.
Task Difﬁculty Tasks should be solvable by most college-educated speakers.
In the meantime, they should be challenging to current models, the performance of which in various test scenarios falls short of human performance.
Task Diversity Tasks should cover a wide range of generation challenges that allow for ﬁndings to be as general as possible.
Input Format The input format of the tasks needs to be as simple as possible to reduce the difﬁculty of data processing.
Besides, it should not contain anything but text (e.g., without any images or videos).
In order to meet the above criteria, 8 domain experts are asked to vote from 10 typical generation tasks 1 .
Finally, four generation tasks are selected for MTG, which are story generation, question generation, title generation and text summarization.
Story generation (SG) aims to generate the end of a given story context, which requires the model to understand the story context and generate a reasonable and ﬂuent ending (Guan et al., 2019).
Question generation (QG) targets at generating a correct question for a given passage and its answer (Duan et al., 2017).
For the same passage with different answers, the system should be able to generate different questions.
Title generation (TG) converts a given article into a condensed sentence while preserving its main idea (Jin and Hauptmann, 2002).
The title should be faithful to the original document and encourage users to read the news
[Table]
Format
Goal
[Table]
Generate
[Table]
SQUAD 1.
Wikipedia
[Table]
Generate
[Table]
question
[Table]
ByteCup
News
[Table] [Table]
>
Generate the title of the document Text Summarization
CNN/DailyMail
News
[Table]
article, summary>
Generate the summary of the document at the same time. Text summarization (Summ) aims to condense the source document into a coherent, concise, and ﬂuent summary (Mani, 2001).
It is similar to title generation but the output of text summarization is relatively longer.
These four tasks focus on different generative abilities and realize different goals.
After conﬁrming the tasks, the next step is to choose the dataset for each task.
The two selection principles are listed as follows:(1) License: Task data must be available under licenses that allow using and redistributing for research purposes.
The dataset should be free and available for download.
(2) Quality:
The dataset size should be as large as possible and the quality should be checked
English datasets are chosen as the initial datasets because they are more accessible in all four tasks and have relatively larger size compared with datasets in other languages.
We choose ROCStories (Mostafazadeh et al., 2016) for story generation, SQUAD 1 .
(Rajpurkar et al., 2016) for question generation, ByteCup 2 for title generation and
CNN/DailyMail
(Nallapati et al., 2016) for text summarization.
These datasets are popular in the corresponding ﬁelds and have been veriﬁed to be high-quality by many works.
Moreover, they are all under a permissive license.
An overview of all task datasets is shown in Table 1.
3. Language Selection
The original datasets are in English (en) only and we want to extend them into a multiway parallel form.
This means that all English texts should be translated into other languages, which will lead to high annotation costs.
Thus, a state-of-the-art translator is leveraged to do the translation and then annotators are asked to correct the translated text.
Considering this construction method, MTG should contain languages that (1) have good English-to-X translators and (2) are diverse in language family.
Finally, German (de), French (fr), Spanish (es) and Chinese (zh) are chosen.
German is from the same language branch as English while French and
Spanish are from different ones.
Chinese is more distant from the rest of languages in the language family tree.
[Table]
3. Data Collection
After determining the tasks and languages, we introduce the data collection process to get the MTG.
The Google Translate 3 is used to translate the English datasets to the selected languages.
To control the quality of translated texts, we back translate the text to English and ﬁlter the samples whose n-gram overlap ratios with the original English texts are lower than a certain threshold.
Different threshold values (from 0 .
3 to 0 .
6 with 0 .
1 as step length) are tested and if it is set to 0 .
6 , the training data size of QG will drop more than 60% .
Thus we decide to use 0 .
5 as the threshold number to improve the quality of the ﬁltered data while still maintaining more than 70% of the original training data.
4 Samples in four languages are aligned to ensure that the dataset is multiway parallel.
20 , 000 samples of each task and language are randomly selected for annotation under the premise
[Table]
of ensuring inter-language alignment. The annotators are required to further check the translated results based on the following rules: (1) Semantic aligned Whether the target text is meaningful and is fully semantic aligned with the source text.
Fluency Whether the translated text is grammatically correct.
(3) Style Whether the translation follows the norms of local culture, language conventions, and gender-related words.
If the translated text contradicts any of the above rules, annotators will correct it accordingly.
The annotated data is then split to 15 k/ 2 k/ 3 k as training/development/test subsets.
A team of 10 full-time experts 5 are hired to do the annotation, who are paid daily.
Some parttime workers 6 are also employed to increase the annotation throughput, who are paid by the number of annotations.
Each annotator is an expert in at least two languages (English and another target language).
They are trained to correct translation errors according to the above rules, ﬁrst a small number of samples for trial, these annotation results are re-checked by us and feedback is given to the annotators to help them understand the tasks better.
After this annotation training process, the annotators start to annotate the dataset.
For quality control, we sample 2% from the annotations and arrange for 9 experts to double-check them.
Each example is assigned to two other experts and the data is qualiﬁed only if both of them agree on the annotation 7 .
If more than 5% of the annotations fail, then all the data of that annotator for that day will be re-checked.
Then the multiway parallel generation benchmark MTG is ﬁnally completed.
It contains four different generation tasks in ﬁve languages and its quality is improved by the incorporation of human annotation.
However, the number of humanannotated data is still small due to cost concerns.
Introducing more human-annotated data or carrying out extra ﬁltering for machine-translated data can be future directions to further improve the quality of MTG.
The statistics of MTG is shown in
Table 2.
Experiments
In this section, we conduct extensive experiments to benchmark the difﬁculty of our proposed MTG via several state-of-the-art multilingual models under different scenarios.
4. Baseline Models
The performance of the following four popular multilingual pretrained models is explored 8 :
M-BERT Multilingual BERT (M-BERT) (Devlin et al., 2019) is a language model pretrained from monolingual corpora in 104 languages
using Masked Language Modeling ( MLM) task.
XLM The Cross-Lingual Language Model (XLM) (Lample and Conneau, 2019) is pretrained with Masked Language Modeling (MLM) task using monolingual data and Translation Language
Modeling (TLM) task using parallel data.
mBART Multilingual BART (mBART) (Liu et al., 2020) is a pretrained encoder-decoder model using denoising auto-encoding objective on monolingual data over 25 languages.
mT 5 Multilingual T 5 (mT 5 ) (Xue et al., 2020) is a multilingual variant of T 5 (Raffel et al., 2020) formatting all tasks as text-to-text generation problems.
mT 5 is pretrained on a span-corruption version of Masked Language Modeling objective over 101 languages.
[Table]
n d e f r e s z h
[Table]
[Figure]
In order to fully understand the model performance, the quality of generated texts is evaluated from different aspects, including metrics measuring the relevance between outputs and references (e.g.,
BLEU, ROUGE, and BERTScore) and metrics measuring the diversity of the generated texts (e.g.,
Distinct).
Moreover, we propose a new ensemble metric leveraging relevance metrics to measure how close the generated text is to human writing.
It not only has higher correlation scores with human judgments but also is capable of measuring model performances fairly between languages.
N-gram based Metrics N-gram based metrics evaluate the text-overlapping scores between the outputs and references.
The following three metrics are used: (1) BLEU (Papineni et al., 2002) is a popular metric that calculates the word-overlap scores between the generated texts and gold-standard ones.
We use the BLEU-4, which is the average score for unigram, bigram, trigram, and 4-gram.
(2) ROUGE (Lin, 2004) is a recall
oriented met - ric that counts the number of overlapping units such as n-gram and word sequences between the produced texts and gold-standard ones.
(3) METEOR (Banerjee and Lavie, 2005) relies on semantic features to predict the similarity scores between system hypotheses and human references.
Embedding based Metrics
The embeddingbased metrics can, to a large extent, capture the semantic-level similarity between the generated text and the ground truth.
BERTScore (Zhang et al., 2019) computes the similarity of candidate and reference as a sum of cosine similarities of tokens using BERT contextual embeddings.
Diversity Metrics We also employ the distinct metric (Li et al., 2016), which calculates the proportion of the distinct n-grams in all the system hypotheses and can be used to evaluate the diversity of the generated texts.
Human Evaluation Human evaluation is also leveraged to better estimate the quality of model outputs.
Speciﬁcally, 30 cases are randomly sampled from the test set for each task and language while ensuring all 30 cases are aligned among ﬁve languages, and then they are presented to human annotators with the model outputs.
The generated texts are evaluated under task-agnostic and taskspeciﬁc aspects.
Task-agnostic aspects include
Grammar , Fluency , Relevance and Language
[Table]
Fusion .
The former three aspects are scored from 1 to 5 while the language fusion score is set toif all tokens of a model-generated text are in the target language and 0 otherwise.
Besides task-agnostic aspects, the generated text is also evaluated under task-speciﬁc aspects.
For title generation and summarization, coverage measures the degree to which the generated text covers the main content of the document.
Correspondence for question generation measures the extent to which the generated question is matched with both document and answer.
For story generation, we further evaluate whether the generated story is logically feasible.
All task-speciﬁc aspects are scored from 1 to 5 .
Ensemble Metric
Some N-gram based metrics such as BLEU and ROUGE largely depend on the tokenizer for speciﬁc languages.
For example, BLEU scores for Chinese outputs are relatively high because it simply uses a character-level tokenizer.
This causes unfair comparison between different languages.
To this end, we propose an ensemble metric that evaluates the degree to which a piece of text resembles manual writing.
It not only enables fair comparison between languages but is also proved to have a better correlation with humanannotated scores at the end of this subsection.
We ﬁrst average the grammar, ﬂuency and relevance scores as targets, then normalize the automatic metrics and human scores among every language to eliminate the score discrepancy between languages.
Three relevance metrics (BLEU, ROUGE-L, and
BERTScore-F1) are gathered as features.
The samples are split into training, development and test sets.
After comparing different regression models’ performance as shown in the upper part of Table 3 , we ﬁnally choose bagging regression model (Breiman, 1996) as the ensemble metric.
Moreover, the bagging ensemble metric shows a higher correlation with human-annotated scores compared with other relevance automatic metrics as shown in the lower part of Table 3.
4. Evaluation Scenarios
To validate the effect of different experimental settings on model performance, several state-of-theart multilingual models are studied under four evaluation scenarios.
Monolingual ﬁne-tuning The pretrained model is tuned for a downstream task using the training data for a speciﬁc language and evaluated on the test set for the same language.
Multilingual ﬁne-tuning The pretrained model is jointly ﬁne-tuned with data in all languages for a speciﬁc task.
Different from the monolingual ﬁnetuning setting, there is only one model for each downstream task, which can serve all languages.
Cross-lingual generation Since MTG is multiway parallel, it can be reorganized to create inputoutput pairs that belong to different languages.
In this paper, we make use of the multiway parallel data to do the supervised cross-lingual training, e.g., for English centric cross-lingual training, we take the English source as the input and the parallel
German, French, Spanish, Chinese target as the out- put.
Then we evaluate the model on same setting (en->de, en->es, en->fr, en->zh).
The cross-lingual generation performances on all 5 ∗ 4 directions are evaluated.
Zero-shot transfer We also try to explore the zero-shot ability of multilingual pretrained models on the four tasks.
The model is ﬁne-tuned on a speciﬁc task with English input and output.
Then it is used to generate output in other languages with a given language tag.
Results 5. Monolingual and Cross-lingual
This section displays the monolingual and crosslingual model comparison to explore their performances in different tasks and languages.
Figure 1 contains the ﬁve language-centric cross-lingual and monolingual results.
Several conclusions can be drawn from the results:
The performance of Cross-lingual is better than monolingual in some cases.
As shown in
Figure 1, model performances on ensemble scores in cross-lingual setting exceed those in monolingual setting frequently (e.g., the monolingual result of French underperforms the English to French cross-lingual result in Figure 1(b) ).
This is because the cross-lingual models are trained with more data (e.g., the English centric cross-lingual model is trained with en->de, en->fr, en->es, en- >zh data), and the data from different cross-lingual directions can sometimes beneﬁt from each other thus improving the model performance.
Chinese text generation is challenging in cross-lingual setting.
As illustrated in Figure 1, nearly all models obtain inferior scores when generating Chinese text.
Also, model results on Chinese inputs are usually worse than results on inputs in other languages.
The wide discrepancies in grammar and vocabulary between Chinese and other languages lead to the poor performance of crosslingual generation when either the target language or source language is Chinese.
Multilingual pretrained models obtain lower scores on the Summarization task.
Compared with other tasks, summarization task requires longer output, which increases the difﬁculty of text generation, thus causing poor performance both in cross-lingual and monolingual settings.
Mono
Multi
Cross
Zero
E n s e m b l e S c o r e
{+Stage
+}
Stage 5. Monolingual and Multilingual In addition to cross-lingual analysis, we also explore the performance difference between models trained in monolingual and multilingual settings. Table 4 displays the monolingual and multilingual training results for four models in four tasks.
In most cases, multilingual training can improve model performance on relevance.
As shown in Table 4, 75 out of 96 multilingual results outperform the monolingual counterparts on various relevance metrics in different tasks.
The reason is that the multilingual data in MTG is fully parallel across all ﬁve languages and every sample has semantically aligned counterparts in other languages.
It makes better semantic fusion among different languages, thus boosting the multilingual training performance.
The advantages of multilingual training are not obvious on diversity measured by distinct1 .
Especially in the story generation task, 3 out of 4 models obtain better distinct-1 scores in monolingual setting than in multilingual one.
Diversity can not be improved by semantic sharing across languages especially when the samples of them are multiway parallel.
This is because the multiway parallel dataset with the semantic aligned samples repeating in different languages encourages models to generate similar texts to some extent.
5. Zero-shot results
To test the cross-lingual generation ability of multilingual pretrained models when no direct crosslingual training data are provided, we evaluate the zero-shot cross-lingual generation performance.
Table 5 presents the zero-shot results for XLM in four tasks.
It demonstrates that the multilin- [Table] gual pretrained model XLM still lacks the ability to generate high-quality cross-lingual output in zero-shot scenario.
Moreover, English to Chinese and French zero-shot generation shows inferior performance.
The performance decline is rather salient when generating Chinese text.
This is because Chinese and French (especially Chinese) are distant from English in the language family tree.
On the other hand, zero-shot results underperform cross-lingual results which further emphasizes the importance of direct cross-lingual training data for cross-lingual text generation.
5. Pseudo and Annotated Data
To answer the question “Does the 400k annotated training data help the model generate better? ”, we use the rough training data ﬁltered by back translation for the ﬁrst stage ﬁne-tuning and the annotated training data for the second stage.
The ablation study results on the two-step ﬁnetuning in summarization under all evaluation scenarios with XLM are illustrated in Figure 2.
The extra human-annotated data boost model performance by at least 3 .
8% on the ensemble metric.
We also make a T-test and prove that the improvement of annotated training data is signiﬁcant in all settings.
10 It demonstrates that although the number of annotated data is small, it can signiﬁcantly improve the performance.
It also highlights the necessity of human-annotated multilingual data compared with pseudo-parallel data via machine translation. [Table]
‘Flu.’, ‘Rel.’
, ‘Lang Fu.’
, ‘Task Spec.’ indicates Grammar ,
Fluency , Relevance , Language Fusion and Task Speciﬁc scores respectively.
5. Human evaluation
Table 6 presents the human evaluation scores for mBART in four tasks.
Multilingual training results can surpass the monolingual results in QG, TG and
Summ on relevance.
In terms of task-speciﬁc score, multilingual results are also superior in SG and
QG.
This is consistent with the conclusion in Sec.
On the other hand, language fusion scores in zero-shot setting are extremely low, indicating the pretrained models still lack the ability to generate texts in correct language in zero-shot setting.
Discussions
Considering the annotation cost, it is not realistic to construct a multiway text generation dataset with all data annotated by human.
As a consequence, most of the non-English data in MTG are automatically translated from their English counterparts.
Although the n-gram consistency check when round-trip translating the data can guarantee the quality of them to some extent, some translation errors are inevitable.
MTG with more annotated data and with data ﬁltered by more reliable methods will be explored in the future.
On the other hand, human often gives an overall evaluation of a generated text rather than measuring it in ﬁne-grained aspects of grammar, ﬂuency and relevance.
Thus we try to propose a metric measuring how a text resembles human writing and consider grammar, ﬂuency and relevance as a whole.
This metric may not be perfect, but it is a promising direction as there does not exist a really reliable text generation metric nowadays.
Conclusion
In this paper, we propose a multilingual multiway benchmark MTG for text generation.
It contains four typical generation tasks: story, question, title generation and text summarization.
The key feature of MTG is that it has multiway parallel data across ﬁve diverse languages: English, German, French,
Spanish and Chinese.
It provides the benchmark with the ability to create cross-lingual data between any two languages and makes the semantic fusion between languages easier.
On the other hand, it provides more evaluation scenarios, such as multilingual training, cross-lingual generation and zeroshot transfer.
We also benchmark state-of-the-art multilingual pretrained models on our MTG from different metrics (including a newly proposed ensemble metric) to explore its features and promote research in multilingual text generation.
Ethics Consideration
Since we propose a new multilingual text generation benchmark MTG, we solve some possible ethic considerations in this section.
English
dataset
We choose
ROCStories,
SQUAD 1.0, ByteCup and CNN/DailyMail as the
English datasets for story, question, title generation and text summarization tasks.
All of them are available for research use under their licenses.
They can be downloaded free from their websites 11 .
We ensure that these datasets are only used for academic research and the dataset construction process complies with the intellectual property and privacy rights of the original authors.
Also, our proposed benchmark suite MTG should only be used for academic research purposes.
Annotation process
As described in Sec. 3.3, we hire some full-time and part-time language experts to do the annotation.
Full-time experts are paid $40 per day and part-time annotators are paid $0 .
2 per example 12 .
Their salary is higher than the local average hourly minimum wage.
All annotators are aware of any risk of harm associated with their participation.
The annotation process is in compliance with the intellectual property and privacy rights of the recruited annotators.
The annotation protocol is proved by the legal department inside the company.
Risk Concern
In this paper we propose a new ensemble metric measuring to what degree is the generated text close to human-level.
The further pursue for more human-like multilingual generation will possibly raise safety concerns.
Bert: Pre-training of
deep bidirectional transformers for language understanding.
In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171–4186.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. Question generation for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 866–874. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu
Anuoluwapo, Antoine Bosselut, Khyathi Raghavi
Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D
Dhole, et al. 2021.
The gem benchmark: Natural language generation, its evaluation and metrics.
arXiv preprint arXiv:2102.01672 .
Jian Guan, Yansen Wang, and Minlie Huang.
Story ending generation with incremental encoding and commonsense knowledge.
In Proceedings of the AAAI Conference on Artiﬁcial Intelligence , volume 33, pages 6473–6480.
Mandy Guo, Zihang Dai, Denny Vrandeˇci´c, and Rami
Al-Rfou. 2020.
Wiki-40b: Multilingual language model dataset.
In Proceedings of The 12th Language Resources and Evaluation Conference , pages 2440–2452.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
Xtreme: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation.
In International Conference on Machine
Learning , pages 4411–4421.
PMLR.
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki,
Haibo Ding, and Graham Neubig. 2020.
X-factr:
Multilingual factual knowledge retrieval from pretrained language models.
In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP)
Liang
Nan Duan
Yeyun Gong
Ning Wu
Fen fei Guo
Weizhen Qi
Ming Gong
Linjun Shou
Cao
Naman Goyal
Xian Li
Edunov
Marjan Ghazvininejad
Lewis
Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu and Lei Li
ACL
IJCNLP 2021 ,
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya
Barua, and Colin Raffel. 2020.
mt
A massively arXiv arXiv
Rong Ye, Wenxian Shi, Hao Zhou
Zhongyu Wei, and
Lei Li.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q
Weinberger, and Yoav Artzi
Bertscore arXiv preprint arXiv:1904.09675 .
Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun
Zhang, Shaonan Wang, and Chengqing Zong
A Back Translation Threshold Testing The detailed data sizes of back translation ﬁltered datasets for different tasks are presented in Table 7.
B Experimental settings
The overall statistics for multilingual pretrained models are presented in Table 8 and the detailed descriptions for them are as follows:
1 M-BERT Multilingual BERT (M-BERT) (Devlin et al., 2019) is a single language model pretrained from monolingual corpora in 104 languages using Masked Language Modeling (MLM) task.
M-BERT leverages a shared vocabulary of 110 k WordPiece tokens and has 12 layers with 172 M parameters totally.
XLM The Cross- Lingual Language Model (XLM) (Lample and Conneau, 2019) is pre-trained simultaneously with Masked Language Model (MLM) task using monolingual data and Translation Language Model (TLM) task using parallel data.
XLM has a shared vocabulary of 200 k byte-pair encoded (BPE) subwords (Sennrich et al., 2016) and 16 layers totaling 570 M parameters.
1 mBART Multilingual BART (mBART) (Liu et al., 2020) is a pre-trained encoder-decoder model using denoising auto-encoding objective on monolingual data over 25 languages.
mBART has a shared vocabulary of 250 k tokens leveraging Sentence Piece tokenization scheme.
mBART consists of 12 -layer encoder and 12 -layer decoder with a total of 680 M parameters.
mT 5 Multilingual T 5 (mT 5 ) (Xue et al., 2020) is a multilingual variant of T 5 (Raffel et al., 2020) leveraging a text-to-text format.
mT 5 is pre-trained on a span-corruption version of Masked Language
Modeling objective over 101 languages.
It is composed of 24 -encoder layers and 24 decoder layers with 13 B parameters.
We use the encoder-decoder architecture for our generation tasks.
Among the models described above, mBART and mT5 have been pretrained for generation tasks, but M-BERT and XLM are only pretrained for encoder representations.
Therefore, we initialize the decoder with the encoder parameters for M-BERT and XLM.
During the pretraining phase, there are no language tags in M-BERT and mT5.
Thus we manually add the language tag at the beginning of the source and target for M-BERT and add the target language tag to the beginning of source for mT5.
We adjust the input format for each task.
For QG, we append the answer to the passage and insert a special token to separate them.
For SG, we take the beginning four sentences as the source and make the last sentence as the target.
We take a two-step ﬁnetuning to make full use of our MTG benchmark.
We ﬁrst use the large rough parallel training data to train our models on the downstream tasks for 20 epochs, and then ﬁnetune the models on the small annotated training data to further improve the generation performance for 10 epochs.
We evaluate the model for everysteps and use the loss on development to choose the best model.
The batch size is 32.
The learning rate and optimizer parameters are set to the default parameters for each model.
All models are trained with 32GB Tesla-V100.
[Table]
[Table]
[Table]
C Signiﬁcant Test Results
The average ensemble metric scores for stage1 and stage2 in four tasks and the corresponding significant test p-values are displayed in Table 9.
As it shows, adding human-annotated training data can always improve the model performance under different settings.
The improvements are signiﬁcant in all settings
D Experimental Results
We present detailed experimental results of our four baseline models under four different evaluation settings here.
[Table]
21.863 de->de 9.
24.97 fr->fr 12.
22.264 es->es 13.
21.207 zh->zh 11.
mT en->en 7.
23.722 de->de 3.
21.368 fr->fr 3.
18.487 es->es 4.
17.837 zh->zh 6.
[Table]
31.694 fr->fr 12.
29.376 es->
[Table]
29.539 zh->
[Table] [Table]
SG
M-BERT en->
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
>
[Table] [Table]
[Table] [Table]
[Table] [Table]
>
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
>
[Table] [Table]
[Table] [Table]
[Table]
XLM
[Table]
>
[Table] [Table]
[Table] [Table]
[Table]
22.182 de
[Table]
>
[Table]
21.8 fr
[Table]
>fr 11.
[Table]
20.087 es->es 11.
20.473 zh
[Table]
>zh 12.
mBART en->en 17.
[Table]
[Table] [Table]
fr
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Summ
[Table] [Table]
[Table] [Table]
XLM
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
339 de
[Table]
>es 1.
[Table] [Table]
[Table] [Table]
864 de->
[Table] [Table]
[Table] [Table]
[Table]
mT de
[Table]
>en
[Table] [Table]
[Table] [Table]
[Table] [Table]
>fr
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Summ
[Table]
BERT
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
XLM
[Table]
en 11.
[Table] [Table]
[Table] [Table]
[Table]
014 de
[Table]
>fr 10.
[Table] [Table]
[Table] [Table]
[Table] [Table]
es 10.
[Table] [Table]
[Table] [Table]
19.705 de
[Table]
>
[Table] [Table]
[Table] [Table]
[Table]
Summ
[Table] [Table]
XLM
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
23.184 es
[Table]
>fr 7.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
mT
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Summ
M
[Table]
BERT
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
XLM
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
>fr 10.
[Table] [Table]
[Table] [Table]
20.617 es
[Table] [Table]
[Table]
>en 15.
31.993 zh->de 1.
23.311 zh->fr 5.
23.772 zh->es 3. mT zh->en 10.
30.058 zh->de 6.
31.781 zh->fr 8.
28.542 zh->es 9.
Summ
M-BERT zh->en 8.
20.782 zh->de 5.
26.296 zh->fr 6.
22.454 zh->es 7.
XLM zh->en 9.
21.267 zh->de 27.
[Table]
[Table] [Table]
es
[Table]
Summ
[Table]
BERT
[Table] [Table]
[Table] [Table]
XLM
[Table] [Table]
[Table] [Table]