Enhanced Sentence Alignment Network for Efficient Short Text Matching (2020)

Abstract: Cross-sentence attention has been widely applied in text matching, in which model learns the aligned information between two intermediate sequence representations to capture their semantic relationship. However, commonly the intermediate representations are generated solely based on the preceding layers and the models may suffer from error propagation and unstable matching, especially when multiple attention layers are used. In this paper, we pro-pose an enhanced sentence alignment network with simple gated feature augmentation, where the model is able to flexibly integrate both original word and contextual features to improve the cross-sentence attention. Moreover, our model is less complex with fewer parameters compared to many state-of-the-art structures. Experiments on three benchmark datasets validate our model capacity for text matching.

## introduction modeling the semantic relationship of a sentence pair is a long standing task in natural language processing, which can be applied in many scenarios such as paraphrase detection and natural language inference @xcite @xcite . neural network approaches have achieved impressive results on solving text matching tasks for the good representation learning ability and benefiting from large datasets @xcite @xcite . one of the major paradigms is attention based neural approach which adopts matching and fusion method @xcite @xcite . specifically, attention mechanism is used as a key component to compute word or phrase alignments between the two parallel sequences, and then the aligned information is fused to update the sentence representations. recent work also adopts multiple matching processes to equip model with power on gradually refining the attention results @xcite @xcite . unfortunately, conducting cross-sentence attention between two intermediate sentence representations may lead to unstable matching since different layers aim at capturing different semantic information @xcite . also, each intermediate representation is highly correlated to the previous layers, and error propagation would affect the following representations and lead to incorrect alignments since model is unable to amend the information without recalling the original semantic features. furthermore, in case of multiple alignment blocks are used, models may suffer from difficulty of training such as vanishing gradients, and low-level features are inefficient to be fully trained. different connection methods are adopted by some recent models to overcome this problem @xcite @xcite . recently pre-trained language models such as bert have achieved impressive improvements on text matching tasks @xcite . despite the promising results, the large parameter size and growing computational requirements make it hard to directly deploy these models to real-time applications @xcite . thus designing efficient and effective models to tackle text matching has been of increasing importance. in this work, we introduce an enhanced sentence alignment network with gated feature augmentation (esan), in which our model integrates the word features (embedding outputs) and contextual features (encoding outputs) to the intermediate representations for each cross-sentence attention, as shown in figure 1 . the embedding outputs contain the original word information, and the encoding outputs represent each token with the aggregated contexts, which are helpful to guide the attention layer to properly capture the aligned information. a gate operation is used to flexibly control how much these two features to be added. also, incorporating the original semantic features directly to the different levels of representation layers can be viewed as a shortcut connection, which is helpful to reduce the training difficulty on lowlevel features. we then apply a simple but effective fusion layer to fuse the aligned features and update the sentence representations gradually. different from previous work @xcite , we do not apply residual connections between alignment layers or use multiple encoders in alignment layers, and our architecture is more efficient and less complex compared with many strong baselines, indicating the feasibility to be deployed in real applications. to demonstrate the effectiveness of our method, we conduct experiments on three text matching datasets: snli, multinli and quora question pairs. the results show our model outperforms strong baselines with fast inference speed. we also conduct model analysis including an ablation study and a case study on attention visualization. ## related work text matching is a key technique for many nlp tasks such as natural language inference @xcite , paraphrase identification @xcite and machine reading comprehension @xcite . as a long standing problem, this area has been in the center of attention and investigated widely. benefiting from large-scale datasets, neural networks have achieved much success for solving this problem. one of the paradigms uses sentence encoding structure, in which two sentences are encoded into vector representations, and then the vectors are combined to make the final prediction @xcite @xcite . however, the interaction of the two input sequences is not directly considered during the encoding process, which makes the model difficult to capture complex relationship. later work adopts matching and aggregation method to model the alignments of the two sentences. @xcite uses a match-lstm to conduct word-level matching of the two sequences. @xcite propose a simple attention operation and use a feed-forward network to integrate the aligned representations. bimpm @xcite uses multi-perspective matching operation to compare two sequences, and applies a bi-lstm network for aggregation. @xcite uses densnet as feature extractor to extract the semantic feature from the interaction tensor. to better capture the sentence alignments in different levels, multiple attention operations can be stacked together. @xcite propose a simple but effective framework with richer alignment features. @xcite leverages multilevel attention refinement component to conduct more extensive matching and improve the results. adin @xcite stacks asynchronous inference layers for a multi-step reasoning process. recently the pre-trained language models have achieved state-of-the-art results on text matching tasks with pre-training and finetuning procedure @xcite . nevertheless, large parameter size and slow inference speed make it hard to directly deploy these structures to the real applications. different from above methods, we propose a simple but effective gated augmentation layer to enrich the intermediate representations with the original word features and contextual features, and thus guide model to produce better alignments. ## conclusions in this work, we present esan, an enhanced sentence alignment network for text matching. we flexibly integrate both word and contextual features to the intermediate representations with a gate operation to conduct better co-attention between two sequences. our model outperforms strong baselines on three datasets and contains fewer parameters, which indicates the model capacity on producing proper alignments for text matching. in the future, we also plan to apply our methods to some other scenarios such as question answering.