Context-Aware Interaction Network for Question Matching (2021)

Abstract: Impressive milestones have been achieved in text matching by adopting a cross-attention mechanism to capture pertinent semantic connections between two sentence representations. However, regular cross-attention focuses on word-level links between the two input sequences, neglecting the importance of contextual information. We propose a context-aware interaction network (COIN) to properly align two sequences and infer their semantic relationship. Specifically, each interaction block includes (1) a context-aware cross-attention mechanism to effectively integrate contextual information when aligning two sequences, and (2) a gate fusion layer to flexibly interpolate aligned representations. We apply multiple stacked interaction blocks to produce alignments at different levels and gradually refine the attention results. Experiments on two question matching datasets and detailed analyses demonstrate the effectiveness of our model.

## introduction semantic text matching is among the most fundamental tasks in natural language processing. given two sentences, the goal is to predict their semantic relationship. in this work, we focus in particular on question matching (qm) benchmarks. recently, the availability of large-scale annotated datasets has led to a proliferation of deep neural architectures for text matching @xcite @xcite . most existing neural models fall into two categories, namely the sentence encoding and the sentence interaction approaches @xcite . the former encodes sentences as fixed-length vector representations, which are then consulted to make the final prediction. the latter considers interactions between two sequences to identify their semantic connections, which tends to yield better results. attention mechanisms are widely adopted for the sentence-interaction approaches, relying on a word-by-word attention matrix to obtain alignment information between two sequences. this has proven fruitful in modeling sentence pair relationships @xcite @xcite . nonetheless, when computing the cross-sentence attention, existing models mostly focus on word-level local matching and fail to fully account for the overall semantics: each value of the attention matrix is based on just two individual tokens from the sequences without full consideration of the context. as shown in figure 1 , in the original attention mechanism, each token individually attends to the other tokens without accounting for important contextual information. however, accurate matching may require a deeper understanding of the two sentences along with pertinent linguistic patterns and constructions @xcite . @xcite show that contextualizing the self-attention network may improve the original representations, but they do not consider the scenario of sentence pairs with crossattention. in this work, we aim to generalize the notion of cross-sentence attention by enabling it to incorporate rich contextual signals. we propose a context-aware interaction network (coin) with a novel context-aware attention layer. this layer enables the model to consult contextual information while computing the cross-attention matrix to measure the word relevance, yielding better contextualized alignments for semantic reasoning. we leverage the self-alignment on each sequence to produce contexts that represent salient features for each token. the subsequent gate fusion layer is designed to enable the model to selectively integrate the aligned representations and control to what extent the new information is to be passed to the following layers, which is similar to a skip connection in mitigating the additional model complexity coming from the deeper structure. finally, an aggregation layer and a multi-head pooling layer are adopted to infer high-level semantic representations for the sequences and predict the result based on the refined representations. to validate the effectiveness of our method, we conduct extensive experiments on the quora and lcqmc datasets, along with further analyses of model components and a case study visualizing the alignment. the results show that by incorporating rich context into cross-attention, our model outperforms state-of-the-art methods without the huge number of model parameters and pre-training on extrinsic data of bert models. ## method question matching can be viewed as a classification task that seeks a label @xmath0, non-duplicate} for a given sentence pair (s a , s b ). figure 2 illustrates our novel sentence interaction approach for this task. in the following, we describe the individual ingredients of this approach. ## conclusion in this work, we propose a context-aware interaction network for question matching. we improve the cross-attention by incorporating contextual cues, and further leverage a gate fusion layer to flexibly integrate the aligned features. experiments on two datasets validate the effectiveness of our architecture and show that accounting for the context enhances the original cross-attention.