Towards Domain-Generalizable Paraphrase Identification by Avoiding the Shortcut Learning (2021)

Abstract: In this paper, we investigate the Domain Generalization (DG) problem for supervised Paraphrase Identification (PI). We observe that the performance of existing PI models deteriorates dramatically when tested in an out-of-distribution (OOD) domain. We conjecture that it is caused by shortcut learning, i.e., these models tend to utilize the cue words that are unique for a particular dataset or domain. To alleviate this issue and enhance the DG ability, we propose a PI framework based on Optimal Transport (OT). Our method forces the network to learn the necessary features for all the words in the input, which alleviates the shortcut learning problem. Experimental results show that our method improves the DG ability for the PI models.

## introduction paraphrase identification (pi) is the task of recognizing whether one text is a restatement of another text, preserving the same meaning while adopting a different expression @xcite . neural network based models have been proposed for the supervised pi task, and achieve decent performance in the single-domain setting @xcite @xcite . at present, the existing pi corpora are restricted to several particular domains @xcite @xcite , while the practical sentence pair for the paraphrase judgment can be from any unlabeled domain. at the same time, building a pi corpus for a novel domain needs massive human effort and is expensive. therefore, a natural question is: for the supervised models trained in the domains with annotated pi corpora, to what extent can they generalize to an out-of-distribution (ood) domain? in this paper, we investigate the multisource @xcite domain generali-zation (dg) @xcite problem for supervised pi. more specifically, we try to learn a pi model based on information from several annotated source domains, and it could generalize well to an unlabeled domain. we investigate several competitive pi models in the dg setting, and observe that their performance deteriorates dramatically when tested in an ood domain. we conjecture that the poor performance is caused by the models' tendency to the shortcut learning @xcite . more specifically, these models are prone to relying on the shortcut features, e.g., some cue words, for classification. these shortcut features are often unique in one particular dataset or domain. when tested on an ood domain, the models' performance deteriorates because the shortcuts are missing. interestingly, this phenomenon is also observed in other nlp tasks or models, such as nli @xcite , reading comprehension @xcite , and bert @xcite . the pi models usually follow the sentence-pair classification paradigm @xcite . some models originally proposed for the other sentence pair classification tasks, e.g., semantic textual similarity (sts) or natural language inference (nli) can be easily adapted to pi. in what follows, we directly apply the suitable models without further clarification. one general character of these models is: they have a component of information aggregation, i.e., the extracted and encoded features are aggregated into one fixed-length vector before computing the loss function. we point out that this step is one cause of the shortcut learning problem. because in this step, it is not uncontrollable that which features should be preserved or discarded in the aggregated vector. inspired by this, we conduct a new design for the final network layer of pi models to improve the dg ## problem formulation the pi corpus is usually organized as a set {((x i , x i ), y i )} n @xmath0. for each tuple ((x, x), y), x and x are two input sentences, the label @xmath1. the associated domain of this dataset is defined as a joint distribution p x y on x × y, where x is the space of input sentence pairs 1 , and y is the label space. then the target of a pi model is to learn a function f : @xmath2, which predicts the label y based on the sentence pair (x, x). we adopt the common setting of multi-source dg as in @xcite . specifically, assume that we can access a set of k(@xmath3. each s k is associated with a distinct joint distribution p k x y , i.e., p k x y ̸ = p k ′ x y , ∀k ̸ = k ′ and k, k ′ ∈ {1, . . . , k}. for s k , the associated dataset contains i.i.d. data {(( the target domain denoted as t is associated with a joint distribution p t x y , where p t x y ̸ = p k x y , ∀@xmath4, . . . , k}. then dg problem for pi is defined as: given the labeled source domains s, we try to learn a model based on information from s such that the model can generalize well to an unseen domain t . it should be noted that dg is more challenging than the related settings such as domain adaptation @xcite or transfer learning @xcite . the difference primarily lies in that dg cannot access both the feature distribution and the label distribution of the the target domain t , which makes it more practical for real-world applications. ## conclusions and future works as a preliminary attempt in this direction, we investigate the dg problem for supervised pi task in this paper. we point out that the aggregation operation is one reason for the poor ood generalization ability of the existing pi models. we incorporate the optimal transport (ot) distance and design a novel classifier layer, i.e., dg-ot layer. it tackles the shortcut learning problem by enforce the network to learn the importance weights and contextualized representations for all the words. the experiments validate the effectiveness of dg-ot layer. avoiding the shortcut learning is only one factor to the dg ability of pi models. another important aspect is how to handle the domain shift, which is left as our future work. besides, sentence pair classification include other tasks such as sts and nli. it is still unclear whether our method is suitable for sts and nli, and we leave this topic as the future work.