Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph (2021)

Abstract: In cross-lingual text classification, it is required that task-specific training data in high-resource source languages are available, where the task is identical to that of a low-resource target language. However, collecting such training data can be infeasible because of the labeling cost, task characteristics, and privacy concerns. This paper proposes an alternative solution that uses only task-independent word embeddings of high-resource languages and bilingual dictionaries. First, we construct a dictionary-based heterogeneous graph (DHG) from bilingual dictionaries. This opens the possibility to use graph neural networks for cross-lingual transfer. The remaining challenge is the heterogeneity of DHG because multiple languages are considered. To address this challenge, we propose dictionary-based heterogeneous graph neural network (DHGNet) that effectively handles the heterogeneity of DHG by two-step aggregations, which are word-level and language-level aggregations. Experimental results demonstrate that our method outperforms pretrained models even though it does not access to large corpora. Furthermore, it can perform well even though dictionaries contain many incorrect translations. Its robustness allows the usage of a wider range of dictionaries such as an automatically constructed dictionary and crowdsourced dictionary, which are convenient for real-world applications.

## introduction modern machine learning methods typically require a large amount of data to achieve desirable performance @xcite @xcite . while such a requirement can be feasible for languages such as english @xcite (i.e., high-resource language), it is not the case for low-resource languages that lack sufficiently large corpora to build reliable statistical models @xcite . because there are more than six thousand languages in the world @xcite , and only a few of them are high-resource, it is important to enable the use of machine learning in low-resource languages. cross-lingual text classification (cltc) is a transfer learning paradigm that aims to incorporate the training data in high-resource languages (i.e., source languages) to solve the classification task in a low-resource language (i.e., target language) more effectively @xcite @xcite . in cltc, it is common to assume that the source and target tasks are identical, and training labeled data in high-resource languages (i.e., source data) are available. such assumptions can be restrictive, and we give the following four examples. first, source data may not be allowed to use owing to the data privacy, e.g., when customers disagree to disclose their opinion to public @xcite @xcite . second, we may not be able to keep the source data, which is also a motivation of sequential transfer learning @xcite . the colossal-size data that are used to train bert @xcite is a good example. the data of this caliber hardly fit in household pc storage. third, it is possible that the target task is quite specific to the target language, which makes it difficult to find the source data in high-resource languages. one example is fake news classification. the news content is highly specific to each region, which may not be reported in a high-resource language. fourth, collecting source labeled data usually requires the prior knowledge of that source language. for example, it is difficult for people who cannot speak chinese to reliably collect data in chinese language. for these reasons, it is beneficial to consider cross-lingual transfer where we do not require task-specific source data in highresource languages. our goal is to overcome the unavailability of task-specific source data by enabling any lowresource languages to utilize high-quality and widely-available resources from any high-resource languages. to achieve this, we design a method that requires only task-independent word embeddings of a source language (e.g. french) and a wordlevel bilingual dictionary (e.g. french-malayalam dictionary) to solve a classification task in the target language. word embeddings can be easily obtained from many sources @xcite @xcite . likewise, bilingual dictionaries are available as man-made commercial products, free lexical database @xcite , or results from dictionary induction algorithms @xcite . the main challenge of our problem is how to utilize source word embeddings and bilingual dictionaries, which is not straightforward for the following reasons. first, given a word, there are many choices of translation. even worse, dictionaries may contain wrong translations, which is often the case for automatically constructed dictionaries. moreover, the compatibility of the source and target languages can be different depending on the context. finally, the quality of source embeddings and bilingual dictionaries can be diverse. this paper aims to design a machine learning method that effectively transfers task-independent embeddings of source languages to task-specific embeddings of the target language. the method should be able to determine the appropriate transfer for each word in the source languages to the translated words in the target language under such circumstances. to solve this problem, we convert bilingual dictionaries into a dictionary-based heterogeneous graph (dhg) that represents words and translations as nodes and edges, respectively. this reduces the problem into graph representation learning, which allows us to use powerful methods, e.g., graph neural networks (gnns), to solve. dhg is heterogeneous because there are many node types (languages) and edge types (language pairs), which are often ignored by gnns in general @xcite @xcite . then, to effectively address the heterogeneity of dhg, we propose dictionary-based heterogeneous graph neural networks (dhgnets) that first aggregate word translations for each language pair (wordlevel aggregation), and then aggregate the results from all languages (language-level aggregation). our contributions can be summarized as follows: first, we propose an alternative solution that enables cross-lingual transfer for text classification by using only 1) task-independent word embeddings of highresource languages and 2) bilingual dictionaries. second, we propose dhg, which can be utilized for cross-lingual transfer by any learning algorithms operating on graphs, such as gnns. third, we propose dhgnets, which effectively uses dhg to solve text classification in a low-resource language. fourth, we provide experimental results to analyze and show the usefulness of the proposed solution and provide extensive analysis of the choice of dictionary, high-resource language, word embeddings, and gnns. the code and resources are available at https://github.com/nutcrtnk/dhgnet . ## related work transfer learning -the goal of transfer learning is to solve a target task with limited target data by incorporating source knowledge from other domains @xcite . the challenge of this problem is how to make use of source knowledge and avoid negative transfer, which is a phenomenon where using source knowledge worsens the performance @xcite . our problem can be categorized as transfer learning where source knowledge does not include source labeled data but only bilingual dictionaries and taskindependent word embeddings of high-resource languages. recently, the transfer learning problem where no source labeled data are provided is called source-free domain adaptation and has been studied extensively in computer vision because of its practicality @xcite @xcite . however, the study of this problem for cross-lingual transfer is limited to the best of our knowledge. note that our problem is significantly different from zero-shot cross-lingual transfer. in that problem, although target data are not available, it is often assumed that source labeled data and the additional target task information are available @xcite @xcite . furthermore, most work in nlp assumes that source and target domains either share the same task @xcite @xcite or language @xcite . @xcite , cltc is the most related problem to ours. however, cltc assumes that source labeled data are available, and the source and target tasks are identical @xcite @xcite @xcite . since the data requirement of cltc can be restrictive, there exist recent methods for weakly supervised cltc where target labels are not required @xcite @xcite . note that source labeled data are still required for such methods. to enable the use of cross-lingual transfer for more applications, our work explores a different direction where source labeled data are unavailable. ## problem formulation in this section, we define our problem formulation. let 𝒳 be an input space and 𝒴 be an output space. without loss of generality, we consider a document classification problem where 𝒳 is a document space and 𝒴 = {1, 2, … , 𝑛 c } is a set of classes, where 𝑛 c denotes the number of classes. in this problem, we have languages ℒ = {ℓ 0 , ℓ 1 , … , ℓ 𝑛 s }, where ℓ 0 is the target language and ℓ 1 , … , ℓ 𝑛 s are source languages. to define a word embedding function, let 𝒱 ℓ be a vocabulary space and 𝑑 ℓ be the dimensionality of the word embeddings of a language ℓ. then, 𝐸 ℓ ∶ 𝒱 ℓ → ℝ 𝑑 ℓ is a word embedding function for a language ℓ. next, we define a bilingual dictionary 𝐷 ℓ 𝑖 →ℓ 𝑗 ∶ 𝒱 ℓ 𝑖 → 2 𝒱 ℓ 𝑗 as a mapping from a known word in a language ℓ 𝑖 to a set of known words in a language ℓ 𝑗 . the following data are provided in this problem setting: the goal of this problem is to learn a classifier 𝑓 ∶ 𝒳 → 𝒴 that optimizes an evaluation metric of interest such as the accuracy or f 1 -measure with respect to the target distribution. 𝐸 ℓ : word embeddings of source language(s) ## proposed method in this section, we propose a novel method for cross-lingual transfer using source word embeddings and bilingual dictionaries. figure 1 illustrates an overview of our proposed method. ## experiments in this section, we conducted experiments to compare our method with baselines and analyze the method under different conditions. dataset -we conducted experiments on four datasets in five different languages. for thai language (th), we used truevoice (thai-t) and wongnai (thai-w) datasets in pythainlp library @xcite . we used news articles in bengali (bn), malayalam (ml), and tamil (ta) languages from indicnlpsuite @xcite . for european languages, we used twitter sentiment @xcite in bosnian (bs) language. in total, we have six different settings. we used the average accuracy and macro-average f 1 -measure of five runs as evaluation metrics. we provide the statistics of each setting in ## discussion and conclusions we proposed a method based on heterogeneous graph neural networks called dhgnet to transfer the source word embeddings through bilingual dictionaries. without task-specific source data, dhgnet demonstrates that it can outperform models pretrained on extremely large corpora. furthermore, our results revealed that dhgnet can perform well even though dictionaries contain many incorrect translations. its robustness opens the possibility to use a wider range of dictionaries such as an automatically constructed dictionary and crowdsourced dictionary. limitation -because our method operates on the word level, some words may not be found in bilingual dictionaries although they are in the dictionaries in a different form. this may occur in languages that have declensions, e.g., malayalam, japanese, and sanskrit. for computational limitation, dhgnet 𝑚𝑢𝑙𝑡𝑖 can be trained with a dhg upto 30k target words 6 , 110k source words, and 1m edges for long documents, e.g., news articles, on a single nvidia tesla v100-16gb.