A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification (2022)

Abstract: We present a multilingual bag-of-entities model that effectively boosts the performance of zero-shot cross-lingual text classification by extending a multilingual pre-trained language model (e.g., M-BERT). It leverages the multilingual nature of Wikidata: entities in multiple languages representing the same concept are defined with a unique identifier. This enables entities described in multiple languages to be represented using shared embeddings. A model trained on entity features in a resource-rich language can thus be directly applied to other languages. Our experimental results on cross-lingual topic classification (using the MLDoc and TED-CLDC datasets) and entity typing (using the SHINRA2020-ML dataset) show that the proposed model consistently outperforms state-of-the-art models.

## introduction in the zero-shot approach to cross-lingual transfer learning, models are trained on annotated data in a resource-rich language (the source language) and then applied to another language (the target language) without any training. substantial progress in cross-lingual transfer learning has been made using multilingual pre-trained language models (plms), such as multilingual bert (m-bert), jointly trained on massive corpora in multiple languages @xcite @xcite . however, recent empirical studies have found that cross-lingual transfer learning with plms does not work well for languages with insufficient pre-training data or between distant languages @xcite , which suggests the difficulty of cross-lingual transfer based solely on textual information. we propose a multilingual bag-of-entities (m-boe) model that boosts the performance of zero- * work done as an intern at studio ousia. we evaluate the performance of the m-boe model on three cross-lingual text classification tasks: topic classification on the mldoc @xcite and ted-cldc (hermann and blunsom, 2014) datasets and entity typing on the shinra2020-ml @xcite dataset. we train the model using training data in the source language (english) and then evaluate it on the target languages. it outperforms our base plms (i.e., m-bert @xcite and the xlm-r model @xcite ) for all target languages on all three tasks, thereby demonstrating the effectiveness of the entity-based representation. furthermore, our model performs better than stateof-the-art models on the mldoc dataset. our contributions are as follows: our method successfully improves the performance on multiple target languages simultaneously without expensive pre-training or additional text data in the target languages. cross-lingual plms zero-shot cross-lingual transfer learning approaches have relied on parallel corpora @xcite or multilingual word representation @xcite . considerable progress has been made on plms for various cross-lingual transfer tasks. the representative models are m-bert @xcite and xlm-r @xcite , which are multilingual extensions of bert @xcite and roberta @xcite , respectively. both models are pre-trained on massive corpora of approximately 100 languages. laser @xcite ) is a plm trained on a parallel corpus of 93 languages by using a sequence-tosequence architecture. improving cross-lingual transfer learning several studies have attempted to improve crosslingual transfer learning by using additional text data in the target language. @xcite proposed using an unlabeled corpus in the target language to bridge the gap between the language and the domain. dong et al. (2020) and keung et al. (2019) incorporated adversarial training using unlabeled target language examples. dong and de melo (2019) and eisenschlos et al. (2019) presented methods for data augmentation in which pseudo-labels are assigned to an unlabeled corpus in the target language. conneau and lample (2019) additionally pre-trained bert-based models using a parallel corpus. however, these methods require extra training on additional text data for each target language, and their resulting models work well only on a single target language. unlike these methods, our method does not require extra training and improves performance simultaneously for all target languages with only a single plm. furthermore, our method can be easily applied to these models since it is a simple extension of a plm and does not modify its internal architecture. enhancing monolingual plms using entities several methods have been proposed for improving the performance of plms through pre-training using entities. ernie (zhang et al., 2019) and knowbert (peters et al., 2019) enrich plms by using pre-trained entity embeddings. luke (yamada et al., 2020b) and eae (févry et al., 2020) train entity embeddings from scratch during pretraining. however, all of these methods are aimed at improving the performance of monolingual tasks and require pre-training with a large corpus, which is computationally expensive. our method dynamically injects entity information into plms during fine-tuning without expensive pre-training. several studies have attempted to incorporate entity information into plms after pre-training to enhance the performance of monolingual tasks. ostendorff et al. (2019) concatenated contextualized representations with knowledge graph embeddings to represent author entities and used them as features for the book classification task. e-bert @xcite inserts kb entities next to the entity names in the input sequence to improve bert's performance for entity-centric tasks. verlinden et al. (2021) introduced a mechanism for combining span representations and kb entity representations within a bilstm-based end-to-end information extraction model. unlike these methods, our method aims to improve the cross-lingual text classification by combining plms with languageagnostic entity embeddings. text classification models using entities several methods have been commonly used to address text classification using entities. explicit semantic analysis (esa) is a representative example; it represents a document as a bag of entities, which is a sparse vector in which each dimension is a score reflecting the relevance of the text to each entity @xcite @xcite . more recently, @xcite proposed cross-lingual explicit semantic analysis (clesa), an extension of esa, to address cross-lingual text classification. clesa computes sparse vectors from the intersection of wikipedia entities in the source and target languages using wikipedia language links. unlike clesa's approach, we address cross-lingual text classification by extending state-of-the-art plms with a language-agnostic entity-based document representation based on wikidata. the most relevant to our proposed approach is the neural attentive bag-of-entities (naboe) model proposed by @xcite . it addresses monolingual text classification using entities as inputs and uses an attention mechanism to detect relevant entities in the input document. our model can be regarded as an extension of naboe by (1) representing documents using a shared entity embedding across languages and (2) combining an entity-based representation and attention mechanism with state-of-the-art plms. ## proposed method figure 1 shows the architecture of our model. the model extracts wikipedia entities, converts them into wikidata entities, and computes the entitybased document representation using an attention mechanism. the sum of the entity-based document representation and the text-based document representation computed using the plm is fed into a linear classifier to perform classification tasks. ## experimental setup in this section, we describe the experimental setup we used for the three cross-lingual text classification tasks. ## results tables 2, 3, and 4 show the results of our experiments. overall, the m-boe models outperformed their baselines (i.e., m-bert and xlm-r) for all target languages on all three datasets. furthermore, there was a significant difference in the mean scores for the target languages for those models in a paired t-test (@xmath0.05). in particular, the performance of our model clearly exceeded that of the m-bert baseline by 2.7% in accuracy, 2.5% in f1, and 2.1% in f1, on the mldoc, ted-cldc, and shinra2020-ml datasets, respectively. for entity typing, using the entities detected with our simple dictionary-based approach achieved comparable performance to using gold entity annotations ( ## analysis we conducted a series of experiments to analyze the performance of our model on the mldoc dataset (