Probing Multilingual Language Models for Discourse (2021)

Abstract: Pre-trained multilingual language models have become an important building block in multilingual Natural Language Processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge across languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.

## introduction large-scale pre-trained neural language models have become immensely popular in the natural language processing (nlp) community in recent years @xcite . when used as contextual sentence encoders, these models have led to remarkable improvements in performance for a wide range of downstream tasks @xcite . in addition, multilingual versions of these models @xcite have been successful in transferring knowledge across languages by providing language-independent sentence encodings. the general usefulness of pre-trained language models has been convincingly demonstrated thanks to persistent creation and application of evaluation datasets by the nlp community. discourse-level analysis is particularly interesting to study, given that many of the currently available models are trained with relatively short contexts such as pairs of adjacent sentences. @xcite use a diverse set of natural language understanding (nlu) tasks to investigate the generality of the sentence representations produced by different language models. @xcite use a broader set of tasks from across the nlp field to investigate the ability of multilingual models to transfer various types of knowledge across language boundaries. our goal in this paper is to systematically evaluate the multilingual performance on nlu tasks, particularly at the discourse level. this combines two of the most challenging aspects of representation learning: multilinguality and discourse-level analysis. a few datasets have been used for this purpose before, most prominently the xnli evaluation set @xcite for natural language inference (nli), and recently also xquad @xcite and mlqa @xcite for question answering (qa). we substantially increase the breadth of our evaluation by adding three additional tasks: we investigate the cross-lingual generalization capabilities of seven multilingual sentence encoders with considerably varying model sizes through their cross-lingual zero-shot performance foot_0 which, in this context, refers to the evaluation scheme where sentence encoders are tested on the languages that they are not exposed to during training. the complied test suite consists of five tasks, covering 22 different languages in total. we specifically focus on zero-shot transfer scenario where a sufficient amount of annotated data to fine-tune a pre-trained language model is assumed to be available only for one language. we believe that this is the most realistic scenario for a great number of languages; therefore, zero-shot performance is the most direct way of assessing cross-lingual usefulness in a large scale. our contributions are as follows: (i) we provide a detailed analysis of a wide range of sentence encoders on large number of probing tasks, several of which have not previously been used with multilingual sentence encoders despite their relevancy, (ii) we provide suitably pre-processed versions of these datasets to be used as a multilingual benchmark for future work with strong baselines provided by our evaluation, (iii) we show that the zero-shot performance on discourse level tasks are not correlated with any kind of language similarity and hard to predict, (iv) we show that knowledge distillation may selectively destroy multilingual transfer ability in a way that harms zero-shot transfer, but is not visible during evaluations where the models are trained and evaluated with the same language. ## background the standard way of training a multilingual language model is through a large non-parallel multilingual corpora, e.g. wikipedia articles, where the models are not provided with any explicit mapping across languages which renders cross-lingual performance of such models puzzling. @xcite and @xcite are the earliest studies to explore that puzzle by trying to uncover the factors that give multilingual bert (henceforth, mbert) its cross-lingual capabilities. @xcite perform a number of probing tasks and hypothesize that the shared sentence pieces across languages gives mbert its generalization ability by forcing other pieces to be mapped into the same space. similarly, @xcite evaluate the performance of mbert in five tasks and report that while mbert shows a strong zeroshot performance, it also retains language-specific information in each layer. @xcite proposes a benchmark to evaluate sentence encoders specifically on discourse level tasks. the proposed benchmark consists of discourse relation classification and a number of custom tasks such as finding the correct position of a randomly moved sentence in a paragraph or determining if a given paragraph is coherent or not. the benchmark is confined to english, hence, only targets monolingual english models. two very recent studies, xtreme @xcite and xglue @xcite , constitute the first studies on the cross-lingual generalization abilities of pre-trained language models via their zero-shot performance. the tasks in both studies largely overlap, where xtreme serves as cross-lingual benchmark consisting of well-known datasets, e.g. xnli, xquad. on the other hand, while covering the most of xtreme tasks foot_1 , xglue offers new datasets which either focus on the relation between a pair of inputs, such as web page-query matching, or on text generation via question/news title generation. in addition to the mbert and certain xlm and xlm-r versions, xtreme includes mmte @xcite whereas xglue evaluates unicoder @xcite among its baselines. ## cross-lingual discourse-level evaluation in discourse research, sentences/clauses are not understood in isolation but in relation to one another. the semantic interactions between these units are usually regarded as the backbone of coherence in various prominent discourse theories including that underlying the penn discourse treebank (pdtb) @xcite , and rhetorical structure theory (rst) @xcite used in the rst discourse treebank @xcite . modelling such interactions requires an understanding that is beyond sentence-level and, from this point-of-view, determining any kind of relation between sentences/clauses can be associated with discourse. although paraphrase detection or natural language inference may not strike as discourse-level tasks at first glance, they both deal with semantic relations between sentences. @xcite show that textual entailment is, in fact, a subclass of restatement relations of the pdtb framework whereas @xcite report an increase in discourse relation classification accuracy when nli is used as the intermediate fine-tuning task. in a similar vein, a stance against a judgement, favor or against, can be seen as contingency: cause: reason and comparison: contrast in pdtb; explanation and antithesis in rst, respectively. therefore, these nlu tasks can be seen as special subsets of discourse relation classification; only a model with a good understanding beyond individual sentences can be expected to solve these tasks. finally, since question answering requires an understanding on discourse level in order to be solved, so we also believe classifying this as a discourse-level task should be uncontroversial. ## experiments we evaluate a wide range of multilingual sentence encoders which learn contextual representations. the evaluated models represent a broad spectrum of model sizes, in order to allow practitioners to estimate the trade-off between model size and accuracy. ## results and discussion we provide an overview of the main results in figure 1 . the detailed results with per-language breakdown are provided in the appendix a. overall, there is a clear difference between the training and zero-shot performance of all models. when averaged over all tasks, the performance loss in zero-shot transfer ranges from 15.58% (xlm-r-large) to 34.96% (distilmbert) which clearly highlights the room for improvement, especially with smaller model sizes. in the rest of the section, we discuss the results in terms of the encoder type, task and the languages. ## conclusion as pre-trained multilingual sentence encoders have become prevalent in natural language processing, research on cross-lingual zero-shot transfer gains increasing importance @xcite . in this work, we evaluate a wide range of sentence encoders on a variety of discourse-level tasks in a zero-shot transfer setting. firstly, we enrich the set of available probing tasks by introducing three resources which have not been utilized in this context before. we systematically evaluate a broad range of widely used sentence encoders with considerably varying sizes, an analysis which has not been made before. the main variable we look at is the performance gap between training-language evaluation and zeroshot evaluation. unsurprisingly, nearly always there is such a gap, but its magnitude depends on a number of factors: these observation provide several starting points for future work: investigating why knowledge distillation seems to hurt zero-shot performance to a much greater extent than same-language sentence encoding ability and what can be done to solve this problem, and explaining the large variations in the zero-shot transfer gap between different discourselevel nlp tasks.