COSY : CO unterfactual SY ntax for Cross-Lingual Understanding (2021)

Abstract: Pre-trained multilingual language models, e.g., multilingual-BERT, are widely used in cross-lingual tasks, yielding the state-of-the-art performance. However, such models suffer from a large performance gap between source and target languages, especially in the zero-shot setting, where the models are fine-tuned only on English but tested on other languages for the same task. We tackle this issue by incorporating language-agnostic information, specifically, universal syntax such as dependency relations and POS tags, into language models, based on the observation that universal syntax is transferable across different languages. Our approach, called COunterfactual SYntax (COSY), includes the design of SYntax-aware networks as well as a COunterfactual training method to implicitly force the networks to learn not only the semantics but also the syntax. To evaluate COSY, we conduct cross-lingual experiments on natural language inference and question answering using mBERT and XLM-R as network backbones. Our results show that COSY achieves the state-of-the-art performance for both tasks, without using auxiliary training data.

## introduction with the emergence of bert @xcite , large-scale pre-trained language models have become an indispensable component in the solutions to many natural language processing (nlp) tasks. recently, large-scale multilingual transformer-based models, such as mbert @xcite , xlm @xcite and xlm-r @xcite , have been widely deployed as backbones in cross-lingual nlp tasks @xcite @xcite . however, these models trained an intuitive way to tackle this is to introduce language-agnostic information-the most transferable feature across languages, which is lacking in existing multilingual language models @xcite . in our work, we propose to exploit reliable language-agnostic informationsyntax in the form of universal dependency relations and universal pos tags @xcite @xcite . as illustrated in figure 1 , the sentences in chinese and english share the same meaning but have differ- factual syntax dependency relation-level counterfactual syntax pos tag-level counterfactual syntax bought i laptops yesterday . new @xmath0. new @xmath1. new @xmath2. the order difference hampers the transferability between english and chinese in conventional language models (with sequential words as input). in contrast, it is clear from figure 1 that the two sentences share identical dependency relations and pos tags. thus, we can incorporate such universal syntax foot_0 information to enhance the transferability across different languages. to achieve this learning objective in deep models, we design syntax-aware networks that incorporate the encodings of dependency relations and pos tags into the encoding of semantics. however, we find that empirically the conventional attention-based incorporation of syntax, e.g., relational graph attention networks @xcite , has little effect on improving the model. one possible reason is that the learning process may be dominated by the pre-trained language models due to their strength in semantic representation learning, which leads to an overfitted model. this raises the question of how to induce the model to focus more on syntax while maintaining its original capability of representing semantics? to this end, we propose a novel counterfactual syntax (cosy) method, inspired by causal inference @xcite and contrastive learning @xcite . the intuition behind cosy is to create copies of training instances with their syntactic features altered (see the "counterfactual" syntax in figure 2 ), and to force the encodings of the counterfactual in-stances to be different from the encodings of their corresponding factual instances. in this way, the model would learn to put more emphasis on the syntactic information when learning how to encode an instance, and such encodings are likely to perform well across languages. we evaluate our cosy method on both question answering (qa) and natural language inference (nli) under cross-lingual settings. experimental results show that, without using any additional data, cosy is superior to the state-of-the-art methods. contributions: 1) we develop a syntax-aware network that incorporates transferable syntax in language models; 2) we propose a novel counterfactual training method that addresses the technical challenge of emphasizing syntax; and 3) extensive experiments on three benchmarks demonstrate the effectiveness of our method for cross-lingual tasks. motivated by the success of multilingual language models on cross-lingual transfer, several works explore how these models work and what their bottleneck is. on the one hand, some studies find that the shared sub-words @xcite and the parameters of top layers @xcite are crucial for cross-lingual transfer. on the other hand, the bottleneck is attributed to two issues: (i) catastrophic forgetting @xcite , where knowledge learned in the pre-training stage is forgotten in downstream fine-tuning; (ii) lack of language-agnostic features @xcite or linguistic discrepancy between the source and the target languages @xcite . in this work, we aim to tackle zero-shot and few-shot cross-lingual transfer by focusing on the second issue. existing works can be roughly divided into two groups. the first proposes to modify the lan-guage model by aligning languages with parallel data @xcite or strengthening sentencelevel representation @xcite . the second group focuses on the learning paradigm for finetuning on downstream tasks. for instance, some methods adopt meta-learning @xcite or intermediate tasks training @xcite to learn cross-lingual knowledge. our cosy belongs to the second group and fills the blank of using the syntactic information in zero-shot (few-shot) cross-lingual understanding. counterfactual analysis. counterfactual analysis aims to evaluate the causal effect of a variable by considering its counterfactual scenario. counterfactual analysis has been widely studied in epidemiology @xcite and social science @xcite . recently, counterfactual reasoning has motivated studies in applications. in the community of computer vision, counterfactual analysis has been successfully applied in explanation @xcite , long-tailed classification @xcite , scene graph generation @xcite , and visual question answering @xcite @xcite . in the community of natural language processing, counterfactual methods are also emerging recently in text classification @xcite , story generation @xcite , dialog systems @xcite , gender bias @xcite , question answering @xcite , and sentiment bias @xcite . to the best of our knowledge, we are the first to conduct counterfactual analysis in cross-lingual understanding. different from previous works @xcite that generate word-level or sentence-level counterfactual samples, our counterfactual analysis dives into syntax level that is more controllable than text and free from complex language generation module. ## cosy: counterfactual syntax cosy aims to leverage the syntactic information, e.g., dependency relations and pos tags, to increase the transferability of cross-lingual language models. specifically, cosy implicitly forces the networks to learn to encode the input not only based on semantic features but also based on syntactic features through syntax-aware networks and a counterfactual training method. as illustrated in figure 3 , cosy consists of three branches with each branch based on syntaxaware networks (san) indicated by a distinct color. the main branch (in black) is the factual branch that uses factual syntax as input. the red and blue branches are counterfactual branches using counterfactual dependency relations and counterfactual pos tags as input, respectively. the counterfactual training method guides the black branch to put more emphasis on syntactic information with the help of other two branches. note that the red and blue branches work for counterfactual training, and only the prediction from the black branch is used in testing. below, we first elaborate the modules of san in section 3.1, and then introduce the counterfactual training method in section 3.2. ## experiments in this section, we evaluate our cosy method for cross-lingual understanding under both zero-shot and few-shot settings. for the zero-shot setting, we use english for training and evaluate the model on different target languages. for the few-shot setting, we follow the implementation in @xcite and use the development set of the target languages for model fine-tuning foot_2 . ## conclusion we study how to effectively plug in syntactic information for cross-lingual understanding. specifically, we propose a novel counterfactual-syntaxbased approach to emphasize the importance of syntax in cross-lingual models. we conduct extensive experiments on three cross-lingual benchmarks, and show that our approach can outperform the sota methods without additional dataset. for future work, we will combine our approach with other orthogonal methods, e.g., meta-learning, to further improve its effectiveness.