Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion (2022)

Abstract: Transformer-based pre-trained models like BERT have achieved great progress on Semantic Sentence Matching. Meanwhile, dependency prior knowledge has also shown general benefits in multiple NLP tasks. However, how to efficiently integrate dependency prior structure into pre-trained models to better model complex semantic matching relations is still unsettled. In this paper, we propose the Dependency-Enhanced Adaptive Fusion Attention (DAFA), which explicitly introduces dependency structure into pre-trained models and adaptively fuses it with semantic information. Specifically, (i) DAFA first proposes a structure-sensitive paradigm to construct a dependency matrix for calibrating attention weights. (ii) It adopts an adaptive fusion module to integrate the obtained dependency information and the original semantic signals. Moreover, DAFA reconstructs the attention calculation flow and provides better interpretability. By applying it on BERT, our method achieves state-of-the-art or competitive performance on 10 public datasets, demonstrating the benefits of adaptively fusing dependency structure in semantic matching task.

## introduction semantic sentence matching (ssm) is a fundamental technology in multiple nlp scenarios. the goal of ssm is to compare two sentences and identify their semantic relationship. it is widely utilized in recommendation systems @xcite , dialogue systems @xcite , search systems @xcite , and so on @xcite . across the rich history of semantic matching research, there have been two main streams of studies for solving this problem. one is to utilize a sentence encoder to convert sentences into lowdimensional vectors in the latent space, and apply a * equal contribution. † corresponding author. parameterized function to learn the matching scores between them @xcite . another paradigm tends to align phrases and aggregate the integrated information into prediction layer to acquire similarity and make a sentence-level decision @xcite . after the emergence of large-scale pre-trained language models (plms), recent work attempts to integrate external knowledge @xcite into plms. for example, sembert @xcite concatenates semantic role annotation to enhance bert. uer-bert @xcite chooses to inject synonym knowledge. syntaxbert @xcite integrates the syntax tree into transformer-based models. meanwhile, leveraging external knowledge to enhance plms has been proven to be highly useful for multiple nlp tasks @xcite . although previous work has achieved great progress in ssm, existing models (e.g., bert, roberta) still cannot efficiently and explicitly utilize dependency structure to identify semantic differences, especially when two sentences are literally similar. to illustrate that, we display an example of misjudgment by bert @xcite in figure 1 . in the first sentence, the dependency between "exceeded" and "company" is obj, between "exceeded" and "apple" is nsubj. its dependency structure is completely opposite to the second sentence. although the literal similarity of these two sentences is extremely high, the semantics are still quite different. based on the above observations, we intuitively believe that the dependency structure needs to be considered in the process of semantic matching. from a deeper perspective, the mlm training approach of most existing plms is optimizing the co-occurrence probability statistically @xcite , but dependency structure can reflect the dependency relationship within the sentence and integrate prior language knowledge to enhance interaction features. combined with the actual attention alignment process, we believe that constructing a dependency matrix, strengthening the attention weight of same dependency and reducing the attention weight of different dependency, will further improve the performance of existing plms. therefore, two systemic questions arise naturally: q1: how to construct a dependency matrix that contains dependency prior knowledge? inconsistent dependency structures can lead to severe semantic divergence even between sentences with similar text. to capture the dependency structure features, we propose a structure-aware paradigm to construct the dependency matrix. our paradigm utilizes three distinct modules, including the dependency similarity between words, the matching of dependency subgraphs, and the tf-idf weights. ## approach in this section, we introduce dafa in detail and the overall architecture is presented in figure 2 . ## qualitative analysis attention distribution and interpretability to verify the calibration effect of the dependency structure alignment as well as to perform a visual interpretability analysis, we display the attention distribution between two sentences that are literally similar but semantically distinct. the weights for one of our attention heads are shown in figure 5 . obviously, vanilla bert is heavily influenced by the same words in sentences. it ignores deep semantic associations and instead over-focuses on shallow literal features, which may lead to wrong predictions. however, after being calibrated by our method, the attention weights not only learn the shared word in sentences, but also pay more attention to the alignment between the dependency structures. for example, dafa not only increases the weight of the same "exceeded" in the two sentences, but also increases the weight between "apple" in the first sentence and "company" in the second one. this is because "apple-exceeded" and "company-exceeded" are subject-predicate dependency structures in the two sentences respectively. meanwhile, attention modules are often used to explore the interpretability of the model @xcite @xcite . in figure 5 (a), we can observe that it is difficult to determine the deep interaction information between sentences by simple contextual features. however, dependency can align sentence structure at the word level. in figure 5 (b), dafa significantly mitigates the strong influence of the same word and enhances the sensitivity to dependency structures. the calibrated attention distribution is more in line with human cognition and validates our methodology. since dependency and semantics are linguistic expressions with different perspectives and granularities, their combination can further improve the model's awareness to discern subtle semantic differences and reduce error propagation problems. in addition, the results of ablation experiments also confirm dafa at each component level and provide results consistent with our predictions. therefore, our method provides better interpretability. ## conclusions in this paper, we propose a dependency-enhanced adaptive fusion attention (dafa), which can adaptively utilize dependency alignment features for semantic matching. based on the context representation capability of bert, dafa enables the model to learn more fine-grained comparison information and enhances the sensitivity of plms to the dependency structure. the experiment results on 10 public datasets indicate that our approach can achieve better performance than multiple strong baselines. since dafa is an end-to-end trained component, it is expected to be applied to other large-scale pre-trained models in the future.