Cross-lingual Text Classification with Heterogeneous Graph Neural Network (2021)

Abstract: Cross-lingual text classification aims at training a classifier on the source language and transferring the knowledge to target languages, which is very useful for low-resource languages. Recent multilingual pretrained language models (mPLM) achieve impressive results in cross-lingual classification tasks, but rarely consider factors beyond semantic similarity, causing performance degradation between some language pairs. In this paper we propose a simple yet effective method to incorporate heterogeneous information within and across languages for cross-lingual text classification using graph convolutional networks (GCN). In particular, we construct a heterogeneous graph by treating documents and words as nodes, and linking nodes with different relations, which include part-of-speech roles, semantic similarity, and document translations. Extensive experiments show that our graph-based method significantly outperforms state-of-the-art models on all tasks, and also achieves consistent performance gain over baselines in low-resource settings where external tools like translators are unavailable.

## introduction the success of recent deep learning based models on text classification relies on the availability of massive labeled data @xcite @xcite . however, labeled data are usually unavailable for many languages, and hence researchers have developed the setting where a classifier is only trained using a resourcerich language and applied to target languages without annotated data @xcite @xcite . the biggest challenge is to bridge the semantic and syntactic gap between languages. most existing methods explore the semantic similarity among languages, and learn a language-agnostic representation for documents from different languages @xcite . this includes recent state-of-the-art multilingual pretrained language models (mplm) @xcite , which pretrain transformer-based neural networks on large-scale multilingual corpora. the mplm methods show superior cross-lingual transfer ability in many tasks @xcite . however, they do not explicitly consider syntactic discrepancy between languages, which may lead to degraded generalization performance on target languages @xcite . on the other hand, there usually exists sufficient unlabeled target-language documents that come naturally with rich information about the language and the task. however, only a handful of previous researches have taken advantage of the unlabeled data @xcite . to integrate both semantic and syntactic information within and across languages, we propose a graph-based framework named cross-lingual heterogeneous gcn (clhg). following the work of textgcn @xcite , we represent all the documents and words as graph nodes, and add different types of information into the graph. we utilize mplm to calculate the representation of all the nodes, and connect documents nodes with semantically similar ones to extract the knowledge in mplm. words are connected with documents based on the co-occurrences as in previous works. however, we choose to separate different word-doc edges by part-of-speech (pos) tags of words to inject some shallow syntactic information into the graph, as pos taggers are one of the most widely accessible nlp tools, especially for low-resource languages. in-domain unlabeled documents are added to the graph if available. to further absorb in-domain language alignment knowledge, we utilize machine translation to create translated text nodes. the text classification task is then formalized as node classification in the graph and solved with a heterogeneous version of graph convolutional networks @xcite . our contributions are summarized as follows: ## related works traditional methods for cross-lingual classification usually translate the texts @xcite or the classifier model @xcite with external aligned resources such as bilingual dictionaries @xcite or parallel corpora @xcite @xcite . recent works focus on learning a shared representation for documents of different languages, including bilingual word embeddings @xcite @xcite , common subword representations @xcite , and multilingual pretrained language models (mplm) @xcite @xcite . in the past few years, graph neural networks (gnn) have attracted wide attention, and become increasingly popular in text classification @xcite @xcite . these existing work mainly focus on monolingual text classification, except a recent work @xcite using meta-learning and graph neural network for cross-lingual sentiment classification, which nevertheless only uses gnn as a tool for meta-learning. ## method in this section, we will introduce our clhg framework, including how to construct the graph and how to solve cross-lingual text classification using heterogeneous gcn. in general, we first construct a cross-lingual heterogeneous graph based on the corpus and selected features, and next we encode all the texts with multilingual pre-trained language models, then we pass the encoded nodes to the heterogeneous gcn, each layer of which performs graph convolution on different subgraphs separated by different edge types, and aggregates the information together. finally, the graph neural network outputs the predictions of doc nodes, which will be compared with groundtruth labels during training. figure 1 shows the overall structure of the framework. ## experiments we evaluate our framework on three different classification tasks, including amazon review sentiment classification @xcite , news category classification from xglue @xcite , and intent classification on a multilingual spoken language understanding (slu) dataset @xcite . more details of each dataset is provided in the appendix. for all the tasks, we use only the english samples for training and evaluate on other 6 languages, which are german (de), french (fr), russian (ru), spanish (es), japanese (ja), and thai (th). ## conclusion in this study, we propose a novel graph-based method termed clhg to capture various kinds of information within and across languages for crosslingual text classification. extensive experiments illustrate that our framework effectively extracts and integrates heterogeneous information among multilingual corpus, and these heterogeneous relations can enhance existing models and are instrumental in cross-lingual tasks. there may exist some better semantic or syntactic features and combinations of features, which we leave as a future work to explore. we also wish to extend our gnn-based framework to different nlp tasks requiring knowledge transfer and adaptation in the future.