Macedon: Minimizing Representation Coding Rate Reduction for Cross-Lingual Natural Language Understanding (2023)

Abstract: Cross-lingual natural language understanding(NLU) is one of the fundamental tasks of NLP. The goal is to learn a model which can generalize well on both high-resource and low-resource language data. Recent pre-trained multilingual language models, e.g., multilingual BERT, XLM, have shown impressive performance on cross-lingual NLU tasks. However, such promising results request the use of sufficient training data, which is a difficult condition to satisfy for low-resource language. When the data is limited in those low resource languages, the accuracy of existing models will drop. In light of this challenge, we investigate the important task of how to train the cross-lingual model with abundant high-source language data and limited low-resource language data. Existing methods typically learn language-agnostic representation via adversarial training and mutual information estimation. Existing approaches may suffer When data is very limited (e.g., low-resource language) because it is challenging to estimate data distribution accurately. To tackle this issue, we propose a conceptually innovative approach to remove language-associated information via m inimizing represent a tion c oding rate r ed ucti on (Macedon). Specifically, Macedon avoids using extra codes to encode language-related information, which is measured by the rate-distortion function. To validate the effectiveness of Macedon, we conduct extensive experiments on three tasks, including paraphrase identification, natural language inference, and query advertisement matching. The experiment results show that the proposed Macedon outperforms state-of-the-art cross-lingual NLU approaches.

## introduction globally, there are about 7,106 living languages spoken 1 , but only 91 languages have at least 10 1 https://www.ethnologue.com/ million first language speakers counted in 2022 foot_0 . therefore, most languages (i.e., low-resource languages) have limited data when considered in natural language understanding (nlu) tasks @xcite , which covers a variety of sub-tasks dealing with machine reading comprehension such as sentiment classification, named entity recognition, and part-of-speech tagging. as a result, models trained on such limited data may have poor performance. an immediate solution one can think of is to collect more labeled data for low-resource languages. however, it could be infeasible due to the prohibitively intensive human annotation process that is both time-consuming and costly. thus, cross-lingual nlu has gained increasing interests in recent years @xcite @xcite , in which the model is trained with abundant high-resource language data and limited annotated low-resource language data to make the model generalize well for all languages. recently, pre-trained multilingual language models (mlms), such as multilingual bert @xcite and xlm @xcite , have shown promising results in transfer learning across different languages in multiple downstream tasks @xcite . in mlms, different languages are able to share one vocabulary table, enabling the model to learn information that is shared among those languages. typical mlms, however, require parallel corpora in order to fine-tune downstream tasks, which can be difficult to achieve when there exist low-resource languages. additionally, it may be difficult to align different languages and yield satisfying transfer performance when there are huge differences between low-resource and high-resource languages in morphology, syntax, or semantics @xcite . prior studies have shown that aligning different language representations is effective for crosslingual transfer. the key idea is to enforce similar representations for sentences that share similar semantics across languages. existing methods fall into two categories. the first line of methods has to be based on parallel corpora. the method aligns different languages in the latent space in accordance with parallel sentences @xcite @xcite @xcite . for languages with limited resources, however, parallel corpora are extremely difficult to obtain. therefore, another line of approaches leverages adversarial training and employs generative adversarial networks (gan) @xcite to learn language-independent representations @xcite @xcite @xcite . it is worthwhile to note that adversarial training-based methods rely on an accurate estimation of data distributions, which may be infeasible with limited labeled examples in low-resource languages. in this paper, we solve the cross-lingual nlu tasks from another perspective. the goal is still to align different languages. however, different from adversarial training, we do not need to align the distributions of different language representations strictly. we propose to relax the condition and simply force different language representations to overlap in the representation space. to achieve this goal, we propose macedon which minimizes representation coding rate reduction. macedon is built upon the coding rate function, which measures the minimal average number of bits to encode a set of vectors @xcite . specifically, we measure two kinds of coding rates: the coding rate of all language representations and the average coding rate of each language representation. we encourage the two coding rates to be almost the same value. intuitively, the learned representation does not use extra codes to encrypt language information, and can be considered as language irrelevant approximately. in addition, the coding rate function uses empirical covariance matrix to compute rate-distortion function directly, which is likely to be singular and highly biased. in macedon, we employ a simple but effective method to modify the empirical covariance matrix to make it positive definite and reduce its bias. the contributions of the paper are summarized as follows: ## experiment in this section, we evaluate the proposed macedonwith the goal of answering the following questions: ## conclusion in this paper, we study a practical scenario, i.e., learning a model with abundant high-resource language data and limited low-resource language data, for cross-lingual nlu tasks. we follow the general idea of aligning different languages, but different from widely used adversarial training models, we propose a novel approach named macedon, which minimizes representation coding rate reduction among different languages. the motivation is to make model not to use extra codes to encode language information. the advantage compared to adversarial training methods is that the proposed macedon does not rely on a large amount of lowresource language data because it does not need to estimate data distribution accurately. we conduct extensive experiments on three public datasets including paraphrase identification, natural language inference, and query advertisement matching. experiment results show that the proposed macedon outperforms the state-of-the-art baselines.