W asserstein Distance Regularized Sequence Representation for Text Matching in Asymmetrical Domains (2020)

Abstract: One approach to matching texts from asymmetrical domains is projecting the input sequences into a common semantic space as feature vectors upon which the matching function can be readily defined and learned. In real-world matching practices, it is often observed that with the training goes on, the feature vectors projected from different domains tend to be indistinguishable. The phenomenon, however, is often overlooked in existing matching models. As a result, the feature vectors are constructed without any regularization, which inevitably increases the difficulty of learning the downstream matching functions. In this paper, we propose a novel match method tailored for text matching in asymmetrical domains, called WD-Match. In WD-Match, a Wasserstein distance-based regularizer is defined to regularize the features vectors projected from different domains. As a result, the method enforces the feature projection function to generate vectors such that those correspond to different domains cannot be easily discriminated. The training process of WD-Match amounts to a game that minimizes the matching loss regularized by the Wasserstein distance. WD-Match can be used to improve different text matching methods, by using the method as its underlying matching model. Four popular text matching methods have been exploited in the paper. Experimental results based on four publicly available benchmarks showed that WD-Match consistently outperformed the underlying methods and the baselines.

## introduction asymmetrical text matching, which predicts the relationship (e.g., category, similarity) of two text sequences from different domains, is a fundamental problem in both information retrieval (ir) and natural language processing (nlp). for example, * corresponding author in natural language inference (nli), text matching has been used to determine whether a hypothesis is entailment, contradiction, or neutral given a premise @xcite . in question answering (qa), text matching has been used to determine whether a answer can answer the given question @xcite . in ir, text matching has been widely used to measure the relevance of a document to a query @xcite . one approach to asymmetrical text matching is projecting the text sequences from different domains into a common latent space as feature vectors. since these feature vectors have identical dimensions and in the same space, matching functions can be readily defined and learned. this type of approach includes a number of popular methods, such as dssm @xcite , de-catt @xcite , cafe @xcite , and re2 @xcite . in real-world matching practices, it is often observed that learning of the matching models is a process of moving the projected feature vectors together in the semantic space. for example, figure 1 shows the distribution of the feature vectors generated by re2. during the training of re2 @xcite on scitail dataset @xcite , it is observed that at the early stage of the training, the feature vectors corresponding to different domains are often separately distributed (according to the visualization by tnse @xcite ) ( figure 1(a) ). with the training went on, these separated feature vectors gradually moved closer and finally mixed together ( figure 1(b) and (c) ). the phenomenon can be explained as follows. given two text sequences from two asymmetrical domains (e.g., nli), the first sequence (e.g., premise) and the second sequence (e.g., hypothesis) are heterogeneous and there exists a lexical gap that needs to be bridged between them (tay et al., 2018c), similar to that of learning crossmodal matching model @xcite . existing studies @xcite have shown that it is essentially critical that the projection network should generate domain-or modal-invariant features. that is, the global distributions of feature vectors should be similar in a common subspace such that their origins cannot be discriminated. the phenomenon is not unique but recurs in the experiments based on other matching models and other datasets. existing text matching models, however, are still lack of constraints or regularizations to ensure that the projected vectors are well distributed for matching. one natural question is, can we design a mechanism that can explicitly guide the mix of the feature vectors and better distribute them. to answer the question, this paper presents a novel learning to match method in which the wasserstein distance (between the two distributions respectively corresponding to the two asymmetrical domains) is introduced as a regularizer, called wd-match. wd-match consists of three components: (1) a feature projection component which jointly projects each pair of text sequences into a latent semantic space, as a pair of feature vectors; (2) a regularizer component which estimates the wasserstein distance with a feed-forward neural network on the basis of the projected features; (3) a matching component which conducts the matching, also on the same set of projected features. the training of wd-match amounts to repeatedly interplays between two branches under the adversarial learning framework: a regularizer branch that learns a neural network for estimating the upper bound on the dual form wasserstein distance, and a matching branch that minimizes a wasserstein distance regularized matching loss. in this way, the minimization of the loss function leads to a learning method not only to minimize the matching loss, but also to well distribute the feature vectors in the semantic space for better matching. to summarize, this paper makes the following main contributions: the source code of wd-match is available at https://github.com/ruc-wsm/wd-match ## related work in this section, we first review the sequence representation used in text matching, then introduce the wasserstein distance and its applications. ## our approach: wd-match in this section, we describe our proposed method wd-match. ## experiments we conducted experiments to test the performances of wd-match, and analyzed the results. algorithm 1 the wd-match algorithm. require: training set @xmath0, y i , z i )} n @xmath1; mini-batch sizes n 1 and n 2 ; adversarial training step k; trade-off coefficient λ; learning rates η 1 and η 2 ; clipping threshold c 1: repeat 2: ## conclusion and future work in this paper, we proposed a novel wasserstein distance-based regularizer to improve the sequence representations, for text matching in asymmetrical domains. the method, called wd-match, amounts to adversarial interplay of two branches: estimating the wasserstein distance given the projected features, and minimizing the wasserstein distance regularized matching loss. we show that the regularizer helps wd-match to well distribute the generated feature vectors in the semantic space, and therefore more suitable for matching. experimental results on four benchmarks showed that wd-match can outperform the baselines including its underlying models. empirical analysis showed the effectiveness of the wasserstein distance-based regularizer in text matching. in the future, we plan to study different regularizers in the asymmetrical text matching task, for further exploring their effectiveness in bridging the gap between asymmetrical domains.