All Information is Valuable: Question Matching over Full Information Transmission Network (2022)

Abstract: Question matching is the task of identifying whether two questions have the same intent. For better reasoning the relationship between questions, existing studies adopt multiple interaction modules and perform multi-round reasoning via deep neural networks. In this process, there are two kinds of critical information that are commonly employed: the representation information of original questions and the interactive information between pairs of questions. However, previous studies tend to transmit only one kind of information, while failing to utilize both kinds of information simultaneously. To address this problem, in this paper, we propose a Full Information Transmission Network (FITN) that can transmit both representation and interactive information together in a simultaneous fashion. More specifically, we employ a novel memory-based attention for keeping and transmitting the interactive information through a global interaction matrix. Besides, we apply an original-average mixed connection method to effectively transmit the representation information between different reasoning rounds, which helps to preserve the original representation features of questions along with the historical hidden features. Experiments on two standard benchmarks demonstrate that our approach outperforms strong baseline models.

## introduction question matching (qm) aims to identify whether two questions have the same intent, which is widely applied in question answering (qa) applications such as community qa and intelligent customer services. typically, qm is regarded as a semantic matching task @xcite . to correctly infer the relationship of a given question pair, there are two kinds of information that should be considered: the representation information of questions that captures the semantics of the texts, and the inter-active information between questions that contains critical hints for relationship reasoning. for better detecting the relationship between question pairs, it's far from being enough to conduct only one single round of reasoning. existing methods commonly resort to multiple interaction modules to do deep reasoning, where each module is generally composed of an encoding layer (can be omitted @xcite ) to update the representation information of questions and an interaction layer for capturing the interactive information between questions @xcite . in such a multi-round reasoning procedure, both the representation and interactive information in history rounds play a vital role in guiding the future inference. however, previous studies either only transmit the representation information @xcite or only the interactive information @xcite , while failing to utilize both kinds of information simultaneously. as shown in figure 1 (i), when performing multiround reasoning, if a model only transmits the representation information, the interactive information between questions will then be simply utilized to generate the representation of questions for future rounds. consequently, the critical hints for relationship reasoning conveyed by interactive information are abandoned and cannot be directly used for future inferences. on the other side, if a model only transmits the interactive information, it is equivalent to conduct multi-round reasoning with only one single pass on question pairs, as shown in @xcite . admittedly, missing the representation information of original questions may lead to understanding deviation and thus bring cascading errors. therefore, as shown in figure 1 (iii), to better perform reasoning between question pairs, a desirable solution should be able to transmit both the representation and interactive information from historical rounds to the current round simultaneously. to address the aforementioned problems, in this paper, we propose a full information transmission network (fitn) that learns to transmit both the representation and interactive information between each round of reasoning. in particular, we propose a novel memory-based attention (mem-att) to transmit the interactive information between question pairs. in the mem-att, we maintain a global interaction matrix as a memory for keeping the interactive information and do inference on top of it. compared with traditional attentions that calculate the alignment score directly, the proposed interaction matrix keeps rich interactive information and is more stable in the update process due to its redundancy. thanks to the global interaction matrix, each round of inference could benefit from the historical interactive information and the whole reasoning procedure is progressive. meanwhile, to effectively transmit the representation information of questions, we introduce an interesting connection method, namely the original-average mixed connection (oa-mixed connection). instead of feeding only the hidden features from the last reasoning round, when performing reasoning at the current round, we regard both the hidden features and the original representation embeddings of questions as the input. such a connection method enables our model an ability to explicitly utilize the entire rich information of original texts when inference. in addition, the oa-mixed connection employs the average operation over hidden features from the last two rounds to build the input hidden feature for the current reasoning round. compared with the residual connection @xcite that treats the information in each round equally, the average connection pays more attention to the information in the nearer rounds, and thus brings better discrimination ability. we evaluate our proposed method on the quora and lcqmc benchmarks. experimental results show that fitn outperforms the non-pretrained baselines with considerable margins. furthermore, compared with pre-trained models (small ones with comparable parameter sizes as fitn), our fitn also achieves better performance, which reveals the advantage of proposed method under resourceconstrained conditions. all these illustrates the effectiveness of our method. in sum, our major contributions are three-fold: ## methodology in this section, we introduce our proposed full information transmission network (fitn) in detail. as shown in figure 2 , fitn comprises three modules: the embedding module, the interaction module and the prediction module. in fitn, we first embed each question in the embedding module, then do inference in the interaction module and finally predict their relationship in the prediction module. we denote two input questions as @xmath0, s 2 , ..., s m } and @xmath1, t 2 , ..., t n } where s i /t j is the i th /j th token of question s/t and m/n is the token length of s/t . ## experiments we evaluate our fitn on two qm benchmarks: the quora (english dataset) @xcite and lcqmc (chinese dataset) @xcite . the quora dataset contains over 400k question pairs collecting from quora, an english community question answering (cqa) website, and the data splits (380k/10k/10k) are provided in bimpm @xcite . the lcqmc collects over 260k question pairs from a chinese cqa website called baiduknows (240k/8k/12k). ## related work question matching can be regarded as a semantic matching task, which core lies in how to model the vector representation of texts @xcite @xcite and reason about the semantic relationship between text pairs. esim @xcite encodes texts through bilstm or treel-stm @xcite and applies the coattention to extract fine-grained alignment information for inference. bimpm @xcite matches texts from multiple perspectives by multiple kinds of attentions. for better inference, many studies tend to employ deeper models. diin @xcite applies a dense-net on the interaction matrix extracted from two texts for deep inference. drcn @xcite iterates one same block multiple times for multi-turn inference. tim-w @xcite is based on deep mutual in-formation estimation. adin @xcite performs multiple rounds of asynchronous reasoning for the nli task. in comparison, our fitn performs better due to the better utilization of historical information. thanks to the knowledge obtained from massive data, pre-trained models can greatly improve the performance of semantic matching, such as bert @xcite albert @xcite . however, the complexity of the model and the time consumption of reasoning are greatly increased, making them not suitable to resourceconstrained scenarios. enhanced-rcnn @xcite compares itself with bert in inference speed and accuracy. although the performance is relative low, its inference speed is 10 times faster than bert-base. under the resourceconstrained condition, directly using publicly available tiny pre-trained models is another solution. these models are commonly pre-trained with large-scale corpus (like albert-tiny and albertbase @xcite )) or distilled from large pretrained models (like bert-tiny @xcite ). compared with these tiny pre-trained models, our fitn achieves better performance. ## conclusion in this paper, we study the task of question matching and propose a full information transmission 1438 network (fitn) that can utilize both the historical representation and the historical interactive information together in a simultaneous fashion. specifically, the fitn employs a memory-based attention to keep and transmit the historical interactive information and an original-average mixed connectivity pattern to transmit the representation information. experimental results on two benchmarks show that our fitn takes advantage of both kinds of information and outperforms strong baselines with considerable margin.