DABERT : Dual Attention Enhanced BERT for Semantic Matching (2022)

Abstract: Transformer-based pre-trained language models such as BERT have achieved remarkable results in Semantic Sentence Matching. However, existing models still suffer from insufficient ability to capture subtle differences. Minor noise like word addition, deletion, and modification of sentences may cause flipped predictions. To alleviate this problem, we propose a novel Dual Attention Enhanced BERT (DABERT) to enhance the ability of BERT to capture fine-grained differences in sentence pairs. DABERT comprises (1) Dual Attention module, which measures soft word matches by introducing a new dual channel alignment mechanism to model affinity and difference attention. (2) Adaptive Fusion module, this module uses attention to learn the aggregation of difference and affinity features, and generates a vector describing the matching details of sentence pairs. We conduct extensive experiments on well-studied semantic matching and robustness test datasets, and the experimental results show the effectiveness of our proposed method.

## introduction semantic sentence matching (ssm) is a fundamental nlp task. the goal of ssm is to compare two sentences and identify their semantic relationship. in paraphrase identification, ssm is used to determine whether two sentences are paraphrase or not @xcite . in natural language inference task, ssm is utilized to judge whether a hypothesis sentence can be inferred from a premise sentence @xcite . in the answer sentence selection task, ssm is employed to assess the relevance between query-answer pairs and rank all candidate answers @xcite . across the rich history of semantic sentence matching research, there have been two main streams of studies for solving this problem. one 1 these authors contributed equally to this work. 2 corresponding author. the secretaries knew the students. the secretaries knew the students slept . is to utilize a sentence encoder to convert sentences into low-dimensional vectors in the latent space, and apply a parameterized function to learn the matching scores between them @xcite . another paradigm adopts attention mechanism to calculate scores between tokens from two sentences, and then the matching scores are aggregated to make a sentence-level decision @xcite . in recent years, pre-trained models, such as bert @xcite , roberta @xcite , have became much more popular and achieved outstanding performance in ssm. recent work also attempts to enhance the performance of bert by injecting knowledge into it, such as sem-bert @xcite , uer-bert @xcite , syntax-bert @xcite and so on. although previous studies have provided some insights, those models do not perform well in distinguishing sentence pairs with high literal similarities but different semantics. figure 1 demonstrates several cases suffering from this problem. although the sentence pairs in this figure are semantically different, they are too similar in literal for those pre-trained language models to distinguish accurately. this could be caused by the self-attention architecture itself. self-attention mechanism focuses on using the context of a word to understand the semantics of the word, while ignoring model-ing the semantic difference between sentence pairs. de-attention @xcite and sparsegen (martins and astudillo, 2016) have proved that equipping with attention mechanism with more flexible structure, models can generate more powerful representations. in this paper, we also focus on enhancing the attention mechanism in transformerbased pre-trained models to better integrate difference information between sentence pairs. we hypothesize that paying more attention to the finegrained semantic differences, explicitly modeling the difference and affinity vectors together will further improve the performance of pre-trained model. therefore, two systemic questions arise naturally: q1: how to equip vanilla attention mechanism with the ability on modeling semantics of fine-grained differences between a sentence pair? vanilla attention, or named affinity attention, less focuses on the fine-grained difference between sentence pairs, which may lead to error predictions for ssm tasks. an intuitive solution to this problem is to make subtraction between representation vectors to harvest their semantic differentiation. in this paper, we propose a dual attention module including a difference attention accompanied with the affinity attention. the difference attention uses subtraction-based cross-attention to aggregate word-and phrase-level interaction differences. meanwhile, to fully utilize the difference information, we use dual-channel inject the difference information into the multi-head attention in the transformer to obtain semantic representations describing affinity and difference respectively. q2: how to fuse two types of semantic representations into a unified representation? a hard fusion of two signals by extra structure may break the representing ability of the pre-trained model. how to inject those information softly to pre-trained model remains a hard issue. in this paper, we propose an adaptive fusion module, which uses an additional attention to learn the difference and affinity features to generate vectors describing sentence matching details. it first inter-aligns the two signals through distinct attentions to capture semantic interactions, and then uses gated fusion to adaptively fuse the difference features. those generated vectors are further scaled with another fuse-gate module to reduce the damage of the pretrained model caused by the injection of difference information. the output final vectors can better describe the matching details of sentence pairs. our main contributions are three fold: ## approach our proposed dabert is a modification of the original transformer structure, whose structure is shown in figure 2 . two submodules are included in this new structure. (1) dual attention module, which uses a dual channel mechanism in multihead attention to match words between two sentences. each channel uses a different attention head to calculate affinity and difference scores separately, and obtains two representations to measure affinity and difference information respectively. (2) adaptive fusion module, which is used to fuse the representation obtained by dual attention. it first uses guide-attention to align the two signals. and then, multiple gate modules are used to fuse the two signals. finally, a vector is output including more fine-grained matching details. in the following sections, we explain each component in detail. ## conclusions in this paper, we propose a novel dual attention enhanced bert (dabert), which can efficiently aggregate the difference information in sentence pairs and soft-integrate it into a pretrained model. based on bert's powerful contextual representation capability, dabert enables the model to learn more fine-grained comparative information and enhances the sensitivity of plms to subtle differences. experimental results on 10 public datasets and robustness dataset show that our method can achieve better performance than several strong baselines. since dabert is an end-to-end training component, it is expected to be applied to other large-scale pre-trained models in the future.