Multilingual and cross-lingual document classification: A meta-learning approach (2021)

Abstract: The great majority of languages in the world are considered under-resourced for successful application of deep learning methods. In this work, we propose a meta-learning approach to document classification in low-resource languages and demonstrate its effectiveness in two different settings: few-shot, cross-lingual adaptation to previously unseen languages; and multilingual joint-training when limited target-language data is available during trai-ing. We conduct a systematic comparison of several meta-learning methods, investigate multiple settings in terms of data availability, and show that meta-learning thrives in settings with a heterogeneous task distribution. We propose a simple, yet effective adjustment to existing meta-learning methods which allows for better and more stable learning, and set a new state-of-the-art on a number of languages while performing on-par on others, using only a small amount of labeled data.

## introduction there are more than 7000 languages around the world and, of them, around 6% account for 94% of the population. 1 even for the 6% most spoken languages, very few of them possess adequate resources for natural language research and, when they do, resources in different domains are highly imbalanced. additionally, human language is dynamic in nature: new words and domains emerge continuously and hence no model learned in a particular time will remain valid forever. with the aim of extending the global reach of natural language processing (nlp) technology, much recent research has focused on the development of multilingual models and methods to efficiently transfer knowledge across languages. meta-learning, or learning to learn @xcite @xcite , is a learning paradigm which focuses on the quick adaption of a learner to new tasks. the idea is that by training a learner to adapt quickly and from a few examples on a diverse set of training tasks, the learner can also generalize to unseen tasks at test time. meta-learning has recently emerged as a promising technique for few-shot learning for a wide array of tasks @xcite @xcite including nlp @xcite . to our best knowledge, no previous work has been done in investigating meta-learning as a framework for multilingual and cross-lingual few-shot learning. we propose such a framework and demonstrate its effectiveness in document classification tasks. the only current study on meta-learning for cross-lingual few-shot learning is the one by @xcite , focusing on natural language inference and multilingual question answering. in their work, the authors focus on applying meta-learning to learn to adapt a monolingually trained classi-algorithm 1 meta-training procedure. require: p(d): distribution over tasks. require: α, β: step size hyper-parameters initialize θ while not done do sample batch of tasks {d l } = {(s l , q l )} ∼ p(d) for all (s l , q l ) do initialize θ our contributions are as follows: 1) we propose a meta-learning approach to few-shot cross-lingual and multilingual adaptation and demonstrate its effectiveness on document classification tasks over traditional supervised learning; 2) we provide an extensive comparison of meta-learning methods on multilingual and cross-lingual few-shot learning and release our code to facilitate further research in the field; foot_0 3) we analyse the effectiveness of meta-learning under a number of different parameter initializations and multiple settings in terms of data availability, and show that meta-learning can effectively learn from few examples and diverse data distributions; 4) we introduce a simple yet effective modification to existing methods and empirically show that it stabilizes training and converges faster to better local optima; 5) we set a new state of the art on several languages and achieve on-par results on others using only a small amount of data. ## meta-learning methods meta-learning, or learning to learn, aims to create models that can learn new skills or adapt to new tasks rapidly from few training examples. unlike traditional machine learning, datasets for either training or testing, which are referred to as metatrain and meta-test datasets, comprise of many tasks sampled from a distribution of tasks p(d) rather than individual data points. each task is asso-ciated with a dataset d which contains both feature vectors and ground truth labels and is split into a support set and a query set, @xmath0, q}. the support set is used for fast adaptation and the query set is used to evaluate performance and compute a loss with respect to model parameter initialization. generally, some model f θ parameterized by θ, often referred to as the base-learner, is considered. a cycle of fast-adaptation on a support-set followed by updating the parameter initialization of the baselearner based on the loss on the query-set is called an episode. in the case of classification, the optimal parameters maximize the probability of the true labels across multiple batches q ⊂ d in few-shot classification/fast learning, the goal is to minimize the prediction error on data samples with unknown labels given a small support set for learning. meta-training (algorithm 1) consists of updating the parameters of the base-learner by performing many of the formerly described episodes, until some stop criterion is reached. following this procedure, the extended definition of optimal parameters is given in eq. 2 to include fast adaptation based on the support set. the underlined parts mark the difference between traditional supervised-learning and meta-learning. the optimal parameters θ * are obtained by solving in this work, we focus on metric-and optimizationbased meta-learning algorithms. in the following sections, their respective characteristics and the update methods in algorithm 1 are introduced. ## data in this section, we give an overview of the datasets we use and the respective classification tasks. @xcite published an improved version of the reuters corpus volume 2 @xcite with balanced class priors for all languages. mldoc consists of news stories in 8 languages: english, spanish, french, italian, russian, japanese and chinese. each news story is manually classified into one of four groups: corporate/industrial, economics, government/social and markets. the train datasets contain 10k samples whereas the test sets contain 4k samples. ## experiments we use xlm-roberta @xcite , a strong multilingual model, as the base-learner in all models. we quantify the strengths and weaknesses of meta-learning as opposed to traditional supervised learning in both a cross-and a multilingual joint-training setting with limited resources. ## results cross-lingual adaptation tables 2 and 3 show the accuracy scores on the target languages on ml-doc and amazon respectively. we start by noting the strong multilingual capabilities of xlm-roberta as our base-learner: adding the full training datasets in three extra languages (i.e., comparing the zero-shot with the non-episodic baseline in the high-resource, 'included' setting) results in a mere 1.2% points increase in accuracy on average for mldoc and 0.6% points for amazon. although the zero-shot foot_1 and non-episodic baselines are strong, in the majority of cases, a metalearning approach improves performance. this holds especially for our version of protomaml (protomamln), which achieves the highest average accuracy in all considered settings. the substantial improvements for russian on mldoc and chinese on amazon indicate that metalearning is most advantageous when the considered task distribution is somewhat heterogeneous or, in other words, when domain drift @xcite is present. for the chinese data used for the sentiment polarity task, the presence of domain drift is obvious as the data is collected from a different website and concerns different products than the other languages. for russian in the mldoc dataset, it holds that the non-episodic baseline has the smallest gain in performance when adding english data (l src ) in the limited-resource setting (0.2% absolute gain as opposed to 5.7% on average for the remaining languages) and even a decrease of 2.4% points when adding english data in the high-resource setting. especially for these languages with domain drift, our version of protomaml (foprotomamln) outperforms the non-episodic baselines with a relatively large margin. for instance, in ## ablations initializing from a monolingual classifier in our experiments, we often assume the presence of a source language (english). we now investigate (in the l @xmath1. we observe that only protonet consistently improves performance, whereas foprotomamln suffers the most with a decrease of 3.1% and 3.96% in accuracy in the low-and high-resource setting respectively. we surmise this difference is attributable to two factors. intuitively, the monolingual classifier aims to learn a transformation from the input space to the final feature space, from which the prototypes for protonet and pro-tomaml are created, in which the learned classes are encoded in their own disjoint sub-spaces such that a linear combination of these features can be used to correctly classify instances. protonet aims to learn a similar transformation, but uses a nearest neighbours approach to classify instances instead. protomaml on the other hand benefits the most from prototypes which can be used to classify instances after the inner-loop updates have been per-formed. this, in combination with the fact that the first-order approximation of protomaml cannot differentiate through the creation of the prototypes, could explain the difference in performance gain with respect to protonet. ## conclusion we proposed a meta-learning framework for fewshot cross-and multilingual joint-learning for document classification tasks in different domains. we demonstrated that it leads to consistent gains over traditional supervised learning on a wide array of data availability and diversity settings, and showed that it thrives in settings with a heterogenous task distribution. we presented an effective adaptation to protomaml and, among others, obtained a new state of the art on german, italian, japanese and russian in the few-shot setting on mldoc.