m PMR : A Multilingual Pre-trained Machine Reader at Scale (2023)

Abstract: We present multilingual Pre-trained Machine Reader (mPMR), a novel method for multilingual machine reading comprehension (MRC)-style pre-training. mPMR aims to guide multilingual pre-trained language models (mPLMs) to perform natural language understanding (NLU) including both sequence classification and span extraction in multiple languages. To achieve cross-lingual generalization when only source-language fine-tuning data is available, existing mPLMs solely transfer NLU capability from a source language to target languages. In contrast, mPMR allows the direct inheritance of multilingual NLU capability from the MRC-style pre-training to downstream tasks. Therefore, mPMR acquires better NLU capability for target languages. mPMR also provides a unified solver for tackling cross-lingual span extraction and sequence classification, thereby enabling the extraction of rationales to explain the sentence-pair classification process.

## introduction multilingual pre-trained language models, acronymed as mplms, have demonstrated strong natural language understanding (nlu) capability in a wide range of languages @xcite @xcite @xcite . in particular, mplms can maintain exceptional cross-lingual language understanding (xlu) capability on unseen target languages though mplms are only fine-tuned on resource-rich source languages like english. it has been proved that optimizing cross-lingual representations of mplms can improve xlu ca-figure 1 : pre-training and fine-tuning of mpmr. pability. for example, cross-lingual supervisions, such as parallel sentences @xcite or bilingual dictionaries @xcite could enhance cross-lingual representations with better language alignment. xlm-r @xcite and mt5 @xcite showed that appropriately incorporating more languages during pre-training leads to better crosslingual representations. a few works enriched the cross-lingual representations with factual knowledge through the utilization of multilingual mentions of entities @xcite and relations @xcite annotated in knowledge graphs. despite their differences, the above methods essentially constructed more diverse multilingual corpora for pre-training mplms. these mplms would presumably meet their saturation points and are known to suffer from curse of multilinguality @xcite @xcite . under this situation, introducing more training data from either existing @xcite or unseen @xcite languages for enhancing mplms may not bring further improvement or even be detrimental to their cross-lingual representations. in the paper, instead of training a new mplm with better cross-lingual representations, we propose multilingual pre-trained machine reader (mpmr) to directly guide existing mplms to perform nlu in various languages. as shown in figure 1 , mpmr resembles pmr @xcite for constructing multilingual machine reading comprehension (mrc)-style data with wikipedia hyperlinks. these data are used to retrofit an mplm into an mpmr through an mrc-style continual pre-training. during retrofitting process (i.e., pretraining), mpmr jointly learns the general sequence classification and span extraction capability for multiple languages. in xlu fine-tuning, mplms solely rely on cross-lingual representations to transfer nlu capability from a source language to target languages. by contrast, mpmr enables the direct inheritance of multilingual nlu capability from the mrc-style pre-training to downstream tasks in a unified mrc formulation, which alleviates the discrepancies between source-language fine-tuning and target-language inference @xcite . therefore, mpmr shows greater potential in xlu than mplms. to improve the scalability of mpmr across multiple languages, we further propose unified q/c construction and stochastic answer position strategies for refining the curation of mrc data. with these two strategies, mpmr can better generalize to low-resource languages and becomes more robust to position bias @xcite . the experimental results show that mpmr obtains clear improvements over xlm-r @xcite on span extraction, with an average improvement of up to 12.6 f1 on tydiqa, and 8.7 f1 on wikiann respectively. the analysis reveals that mpmr benefits from more multilingual mrc data for pre-training. we also found that mpmr converges faster in downstream tasks and is capable of using its strong extraction capability for explaining the sequence classification process. ## mpmr we present the mrc model and training data of mpmr. we closely follow pmr @xcite and introduce the modifications for enabling multilingual mrc-style pre-training. ## experimental setup implementation details. in mpmr, the encoder is loaded from xlm-r @xcite ) and the extractor is randomly initialized. both components are then continually pre-trained using the multilingual mrc data that we constructed. more hyper-parameters can be found in appendix a.1. downstream xlu tasks. we evaluated mpmr on a series of span extraction tasks, including extractive question answering (eqa), named entity recognition (ner), and aspect-based sentiment analysis (absa). we also evaluated our mpmr on two sequence classification tasks. we followed @xcite to convert all tasks into mrc formulation to effectively leverage the knowledge that is acquired during mrc-style pre-training. for eqa, we used xquad @xcite , mlqa @xcite , and tydiqa @xcite . for ner, we used wikiann @xcite and conll @xcite . semeval16 @xcite was used for absa task. regarding the sequence classification, we used xnli @xcite and paws-x @xcite . additional dataset information and concrete examples are provided in appendix a.2 baselines. we compared mpmr with recent methods on improving cross-lingual representations, including 1) models pre-trained on a large number of languages: xlm-r @xcite , mt5 @xcite , and veco @xcite ; 2) models that exploited multilingual entity information: wiki-cl @xcite , and mluke-w @xcite ; and 3) model that utilized multilingual relation information: kmlm @xcite . for a fair comparison, all models have approximately the same parameter size. to reflect the impact of our mrc-style data and stochastic answer position method on pre-training, we present a stepby-step analysis of the retrofitting process starting from xlm-r in explainable sentence-pair classification. inspired by pmr @xcite , we investigated if the extraction capability of mpmr can be leveraged to explain sentence-pair classification. note that sentence-pair classification focuses on the inference between the two sentences. if we construct the query with only the task label as pmr does, such query does not solely correspond to any meaningful span in the context, and thus is hard to guide the span extraction. therefore, we leveraged another template "[cls] label sen-1 [sep] sen-2 [sep]", where the two sentences are represented separately in the query and the context. in this template, we can extract the exact span from sen-2 that leads to a contraction or entailment relation (i.e., the task label) with sen-1. specifically, we passed the sentence pair to the model twice, with each sentence of the pair being designated as the sen-2 respectively, and extract the context span with the highest probability score from both sentences. as shown in ## conclusions this paper presents a novel multilingual mrc-style pre-training method, namely mpmr. mpmr provides a unified solver for cross-lingual span extraction and sequence classification and enables direct transfer of nlu capability from pre-training to downstream tasks. mpmr clearly improves the previous baselines and provides a possible solution to explain the sentence-pair classification process.