Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents (2022)

Abstract: Text semantic matching is a fundamental task that has been widely used in various scenarios, such as community question answering, information retrieval, and recommendation. Most state-of-the-art matching models, e.g., BERT, directly perform text comparison by processing each word uniformly. However, a query sentence generally comprises content that calls for different levels of matching granularity. Specifically, keywords represent factual information such as action, entity, and event that should be strictly matched, while intents convey abstract concepts and ideas that can be paraphrased into various expressions. In this work, we propose a simple yet effective training strategy for text semantic matching in a divide-and-conquer manner by disentangling keywords from intents. Our approach can be easily combined with pre-trained language models (PLM) without influencing their inference efficiency, achieving stable performance improvements against a wide range of PLMs on three benchmarks.

## introduction text semantic matching aims to predict a matching category or a matching score reflecting the semantic similarity given a pair of text sequences, which is a fundamental task employed in a wide range of applications @xcite @xcite @xcite . recently, pre-trained language models (plm) show remarkable capability of representation learning and have accelerated the research of text semantic matching @xcite @xcite . they typically exploit large-scale corpora and well-designed self-supervised learning objectives to better learn semantic representations, achieving state-of-theart performances or even surpassing the level of non-expert humans on general semantic matching benchmarks @xcite . most existing plms aim to establish a foundation for various downstream tasks @xcite and focus on finding a generic way to encode text sequences. when applied to the task of text semantic matching, it is a common practice to add a simple classification objective for fine-tuning and directly perform text comparison by processing each word uniformly. nevertheless, each sentence can be typically decomposed into content with different levels of matching granularity @xcite . exemplar sentence pairs can be found in figure 1 . the primary content refers to keywords that reflect the factual information about entities or actions, which should be strictly matched. the other content constitute abstract intents, which can be generally paraphrased into various expressions to convey the same concepts or ideas. considering the situation where sentence content has different levels of matching granularity, we propose dc-match, a simple but effective training regime for text semantic matching in a divide-andconquer manner. specifically, we break down the matching problem into two sub-problems: keyword matching and intent matching. given a pair of input text sequences, the model learns to disentangle keywords from intents by utilizing the method of distant supervision. in addition to the standard se-quence matching that has a global receptive field, we further match keywords and intents separately to learn the way of content matching under different levels of granularity. finally, we design a special training objective that combines the solutions to the sub-problems, which minimizes the kl-divergence between the global matching distribution (original problem) and the joint keyword-intent matching distribution (sub-problems). at inference time, we expect that the global matching model automatically distinguishes keywords from intents, then makes final predictions conditioned on the disentangled content in different matching levels. we adopted dc-match to a wide range of plms. comprehensive experiments were conducted on two english text matching benchmarks mrpc @xcite and qqp @xcite , and a chinese benchmark medical-sm. our approach can be easily combined with plms plus few additional parameters, but still achieves stable performance improvements against most baseline plms. notably, all the auxiliary procedures and parameters are only involved in the training stage. the inference efficiency of our approach is exactly the same as that of plm baselines, without additional parameters and computations. our codes and datasets are publicly available foot_0 . our contributions are three-fold: 1) we introduce a novel training regime for text matching, which disentangles keywords from intents based on different levels of matching granularity in a divide-and-conquer manner. 2) the proposed approach is simple yet effective, which can be easily combined with plms plus few auxiliary training parameters while not changing their original inference efficiency. 3) experimental results on three benchmarks across two languages demonstrate the effectiveness of our approach in different aspects. ## related work text semantic matching plays an important role in many applications, such as information retrieval (ir) and natural language inference (nli). traditional technologies exploit neural networks with different inductive biases, e.g., cnn @xcite , @xcite , gnn @xcite , and attention mechanism @xcite . to enhance the matching performance, dozens of works use richer syntactic or hand-crafted features @xcite @xcite , add complex alignment computations @xcite @xcite , and perform multipass matching procedures @xcite , which shows the effectiveness of representation-oriented approaches and model designing strategies based on information interaction. recently, large-scale pre-trained language models (plm) have boosted the performance of text semantic matching by making full use of massive text resources. most of them are composed of multiple transformer layers @xcite with multi-head attentions and are pre-trained with well-designed self-supervised learning objectives. representative models like bert @xcite , roberta @xcite , and albert @xcite aim to establish a powerful encoder that has a comprehensive understanding of input texts. for the task of text semantic matching, plms can be fine-tuned under a paradigm of sequence classification with only an additional classification layer, achieving state-of-the-art performances on general semantic matching benchmarks @xcite . plms can be regarded as foundation models @xcite and they mainly focus on finding a generic way to encode text sequences. instead of processing each word uniformly, in this work, we devise a novel training regime that processes sentence pairs by disentangling keywords from intents, which can be easily combined with plms to stack additional improvements to text semantic matching. ## methodology in this section, we detail the proposed training regime dc-match. it consists of three training objectives: a classification loss for the global matching model; a distantly supervised classification loss that learns to distinguish keywords from intents; a special training objective following the idea of divide and conquer, which uses the kl-divergence to ensure that the global matching distribution (original problem) is similar to the distribution of combined solutions to disentangled keywords and intents (sub-problems). the overall framework is illustrated in figure 2 . ## conclusion in this work, we devise a divide-and-conquer training strategy dc-match for text semantic matching. it breaks down the matching problem into two subproblems: keyword matching and intent matching. the model learns to disentangle keywords from intents that require different levels of matching granularity. the proposed dc-match is simple and effective, which can be easily combined with plms plus few additional parameters. we conduct experiments on three text matching datasets across different languages. experimental results show that our approach can not only achieve stable performance improvement, but also shows robustness to semantically invariant text transformations.