Rationalizing Text Matching: L earning Sparse Alignments via Optimal Transport (2020)

Abstract: Selecting input features of top relevance has become a popular method for building self-explaining models. In this work, we extend this selective rationalization approach to text matching, where the goal is to jointly select and align text pieces, such as tokens or sentences, as a justification for the downstream prediction. Our approach employs optimal transport (OT) to find a minimal cost alignment between the inputs. However, directly applying OT often produces dense and therefore uninterpretable alignments. To overcome this limitation, we introduce novel constrained variants of the OT problem that result in highly sparse alignments with controllable sparsity. Our model is end-to-end differentiable using the Sinkhorn algorithm for OT and can be trained without any alignment annotations. We evaluate our model on the StackExchange, MultiNews, e-SNLI, and MultiRC datasets. Our model achieves very sparse rationale selections with high fidelity while preserving prediction accuracy compared to strong attention baseline models.

## introduction the growing complexity of deep neural networks has given rise to the desire for self-explaining models @xcite @xcite @xcite @xcite . in text classification, for instance, one popular method is to design models that can perform classification using only a rationale, which is a subset of the text selected from the model input that fully explains the model's prediction @xcite @xcite . this selective rationalization method, often trained to choose a small yet sufficient number of text spans, makes it easy to interpret the model's prediction by examining the selected text. in contrast to classification, very little progress has been made toward rationalization for text matching models. the task of text matching encompasses a wide range of downstream applications, such as similar document recommendation @xcite , question answering @xcite , and fact checking @xcite . many of these applications can benefit from selecting and comparing information present in the provided documents. for instance, consider a similar post suggestion in a tech support forum as shown in figure 1 . the extracted rationales could provide deeper insights for forum users while also helping human experts validate and improve the model. in this work, we extend selective rationalization for text matching and focus on two new challenges that are not addressed in previous rationalization work. first, since text matching is fundamentally about comparing two text documents, rationale selection should be jointly modeled and optimized for matching. second, the method should produce an interpretable alignment between the selected rationales showcasing their relations for the downstream prediction. this is very different from rationaliza-tion for text classification, where the selection is performed independently on each input text and an alignment between rationales is unnecessary. one popular method for aligning inputs is attention-based models @xcite @xcite @xcite . however, a limitation of neural attention is that the alignment is rarely sparse, thus making it difficult to interpret how the numerous relations among the text spans lead to the model's prediction. recent work has explored sparse variants of attention @xcite @xcite @xcite , but the number of non-zero alignments can still be large @xcite . additionally, because of the heavy non-linearity following most attention layers, it is difficult to truly attribute the model's predictions to the alignment, which means that attention-based models lack fidelity. we propose to address these challenges by directly learning sparse yet sufficient alignments using optimal transport (ot). we use ot as a building block within neural networks for determining the alignment, providing a deeper mathematical justification for the rationale selection. in order to produce more interpretable rationales, we construct novel variants of ot that have provable and controllable bounds on the sparsity of the alignments. selecting and aligning text spans can be jointly optimized within this framework, resulting in optimal text matchings. our model is fully end-to-end differentiable using the sinkhorn algorithm @xcite for ot and can be used with any neural network architecture. we evaluate our proposed methods on the stackexchange, multinews @xcite , e-snli @xcite , and mul-tirc @xcite datasets, with tasks ranging from similar document identification to reading comprehension. compared to other neural baselines, our methods show comparable task performance while selecting only a fraction of the number of alignments. we further illustrate the effectiveness of our method by analyzing how faithful the model's predictions are to the selected rationales and by comparing the rationales to humanselected rationales provided by @xcite on the e-snli and multirc datasets. alignment. models can be made more interpretable by requiring that they explicitly align related elements of the input representation. in nlp, this is often achieved via neural attention @xcite @xcite @xcite . many variants of attention, such as temperature-controlled attention @xcite and sparsemax @xcite , have been proposed to increase sparsity within the attention weights. however, it is still debatable whether attention scores are truly explanations @xcite . distance-based methods of aligning text have also been proposed @xcite , but they similarly cannot guarantee sparsity or explainability. in this work, we explicitly optimize rationale selection and alignment as an integral part of the model and evaluate the degree to which the alignment explains the model's predictions. ## problem formulation consider two related text documents d x and d y . these documents are broken down into two sets of text spans, s x and s y , where the text spans can be words, sentences, paragraphs, or any other chunking of text. these text spans are then mapped to vector representations using a function g(•) (e.g., a neural network), which produces two sets of vectors representing the inputs, we define an interpretable text matching as an alignment between the text spans in x and y that explains the downstream prediction. following common practice for previous self-explaining models @xcite , we specify that a desirable model must produce alignments satisfying the following criteria of interpretability. explicitness. the alignment between text spans generated by the model should be an observable and understandable component of the model. our model explicitly encodes the alignment between x and y as a matrix @xmath0,j indicates the degree to which x i and y j are aligned. ## background: optimal transport an instance of the discrete optimal transport problem consists of two point sets, @xmath1, with x i , y @xmath2. additionally, x and y are associated with probability distributions @xmath3, respectively, where σ n is the probability simplex σ n := @xmath4: n @xmath5. a cost function c(x, y) : r d × r @xmath6. the costs of aligning all pairs of points are summarized by the cost matrix @xmath7, where c i,@xmath8, y j ). the goal of optimal transport is to compute a mapping that moves probability mass from the points of x (distributed according to a) to the points of y (distributed according to b) so that the total cost of moving the mass between points is minimized according to the cost function c. this mapping is represented by a transport plan, or alignment matrix, @xmath9, where p i,j indicates the amount of probability mass moved from x i to y j . the space of valid alignment matrices is the set u(a, b) := {@xmath10: p1 @xmath11, p t 1 @xmath12. under this formulation, the optimal transport problem is to find the alignment matrix p that minimizes the sum of costs weighted by the alignments: l c (a, b) := min @xmath13,b) c, @xmath14,j c i,j p i,j . note that this optimization is a linear programming problem over the convex set u(a, b). as a result, one of the extreme points of u(a, b) must be an optimal solution. ## learning interpretable alignments using "vanilla" ot produces sparse alignments as guaranteed by proposition 1, but the level of sparsity is insufficient to be interpretable. for instance, alignment 1 in figure 2 still has a significant number of non-zero alignment values. motivated by this limitation, we propose to encourage greater sparsity and interpretability by constructing ot problems with additional constraints. ## experimental setup datasets. we evaluate our model and all baselines on four benchmarks: two document similarity tasks, multinews and stackexchange, and two classification tasks, e-snli and multirc. the e-snli and multirc tasks come from the eraser benchmark @xcite , which was created to evaluate selective rationalization models. we chose those two datasets as they are best suited for our text matching setup. stackexchange foot_1 is an online question answering platform and has been used as a benchmark in previous work @xcite @xcite . we took the june 2019 data dumps foot_2 of the askubuntu and su-peruser subdomains of the platform and combined them to form our dataset. multinews @xcite ) is a multidocument summarization dataset where 2 to 10 news articles share a single summary. we consider every pair of articles that share a summary to be a similar document pair. multirc @xcite is a reading comprehension dataset with the goal of assigning a label of true or false to a question-answer pair depending on information from a multi-sentence document. we treat the concatenated question and answer as one input and the document as the other input for text matching. human rationales are provided as highlighted sentences in the document. for stackexchange and multinews, we split the documents into 80% train, 10% validation, and 10% test, while for e-snli and multirc, we use the splits from @xcite . metrics. we evaluate models according to the following three criteria. 1. sparsity. to evaluate sparsity, we compute the average percentage of active alignments produced by each model, where an alignment is active if it exceeds a small threshold λ. this threshold is necessary to account for numerical imprecision in alignment values that are essentially zero. we set λ = 0.01 n×m unless otherwise specified, where n and m are the number of text spans in the two documents. baselines and implementation details. we use the decomposable attention model @xcite as our baseline attention model. in addition, we compare our model to two attention variants that are designed to encourage sparsity. the temperature attention variant applies a temperature term t in the softmax operator @xcite . the sparse attention variant adopts the sparsemax operator @xcite in place of softmax to produce sparse attention masks. our constrained ot model operates as illustrated in figure 4 . after splitting the input documents into sentences, our model independently encodes each sentence and computes pairwise costs between the encoded representations foot_3 . dummy and replica encodings are added as needed for the desired type of constrained alignment. our model then applies ot via the sinkhorn algorithm to the cost matrix c to produce an optimal alignment matrix p. for the document ranking tasks, the final score is simply c, p . for the classification tasks, we use the alignment p as a sparse mask to select encoded text representations, and we feed the aggregated representation to a shallow network to predict the output label, similar to our baseline attention models. for a fair comparison, our models and all baselines use the same neural encoder to encode text spans before the attention or ot operation is applied. specifically, we use roberta @xcite , a state-of-the-art pre-trained encoder, for the stackexchange and multirc dataset. we use use bi-directional recurrent encoders @xcite for the multinews and e-snli datasets foot_4 . the value of k for the ot constraints is chosen for each dataset by visually inspecting alignments in the validation set, though model performance is robust to the choice of k. in order to compare our models' rationales to human annotations, we use a binary thresholding procedure as described in appendix c. we report results averaged over 3 independent runs for each model. additional implementation details are provided in appendix c. ## results synthetic visualizations. before experimenting with the datasets, we first analyze the alignments obtained by different methods on a synthetic cost matrix in figure 5 . as shown in the figure, all attention baselines struggle to produce sufficiently sparse alignments, even with the use of a small temperature or the sparsemax operator. in contrast, our methods are very sparse, as a result of the provable sparsity guarantees of the constrained alignment problem. for instance, the relaxed one-to-k assignment produces fewer active alignments than either the number of rows or columns, and the exact-k assignment finds exactly @xmath15. ## conclusion balancing performance and interpretability in deep learning models has become an increasingly important aspect of model design. in this work, we propose jointly learning interpretable alignments as part of the downstream prediction to reveal how neural network models operate for text matching applications. our method extends vanilla optimal transport by adding various constraints that produce alignments with highly controllable sparsity patterns, making them particularly interpretable. our models show superiority by selecting very few alignments while achieving text matching performance on par with alternative methods. as an added benefit, our method is very general in nature and can be used as a differentiable hard-alignment module in larger nlp models that compare two pieces of text, such as sequence-to-sequence models. furthermore, our method is agnostic to the underlying nature of the two objects being aligned and can therefore align disparate objects such as images and captions, enabling a wide range of future applications within nlp and beyond. model task f1 % token r. f1 ot (1:1) 59.5 20.3 24.2 ot (1:2) 60.1 28.0 26.5 ot (relaxed 1:1) 59.7 13.6 19.5 ot (relaxed 1:2) 60.2 24.7 29.1 ot (exact @xmath16.0 15.2 22.7 attention 61.4 33.2 15.7 attention (@xmath17.1) 61.0 34.7 17.5 attention (@xmath18.01) 61.0 34.4 18.5 sparse attention 60.7 37.5 25.0 ot (1:1) (+s) 62.1 20.5 48.1 ot (1:2) (+s) 60.0 31.3 46.0 ot (relaxed 1:1) (+s) 60.3 18.2 46.2 ot (relaxed 1:2) (+s) 60.6 25.2 44.9 ot (exact @xmath19.2 16.7 48.7