Enhancing Dual- E ncoders with Question and Answer Cross- E mbeddings for Answer Retrieval (2021)

Abstract: Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. Currently most conventional Dual-Encoders learn the semantic representations of questions and answers merely through matching score. Researchers proposed to introduce the QA interaction features in scoring function but at the cost of low efficiency in inference stage. To keep independent encoding of questions and answers during inference stage, variational auto-encoder is further introduced to reconstruct answers (questions) from question (answer) embeddings as an auxiliary task to enhance QA interaction in representation learning in training stage. However, the needs of text generation and answer retrieval are different, which leads to hardness in training. In this work, we propose a framework to enhance the Dual-Encoders model with question answer cross-embeddings and a novel Geometry Alignment Mechanism (GAM) to align the geometry of embeddings from Dual-Encoders with that from Cross-Encoders. Extensive experimental results show that our framework significantly improves Dual-Encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets.

## introduction answer retrieval @xcite is an important mechanism in question answering (qa) systems to obtain answer candidates given a new question. currently, the most widely used framework for answer retrieval task is dual-encoders @xcite @xcite , also known as "siamese network" @xcite . the dual-encoders model consists of two encoders to compute the embeddings of questions and answers independently, and also a predictor to estimate the relevance by a similarity score between the two embeddings. recently, due to the application of advanced encoding techniques, e.g., transformer @xcite , bert @xcite , the dual-encoders achieved a huge boost on the overall performance @xcite . however, there remains some room to improve since the embeddings of questions and answers are encoded separately, while the cross information between questions and answers are important for answer retrieval @xcite . many efforts have been devoted in developing more powerful scoring by considering the interactions among questions and answers. for example, @xcite introduced additional wordlevel interaction features between questions and answers for matching degree estimation. similarly, @xcite implemented attention mechanism to extract more information when computing matching score. though such approaches improve the scoring mechanism, the overall efficiency derived from separate and off-line embeddings of questions and answers is sacrificed to some extent. therefore, it deserves discussing how to achieve better trade-off for maintaining the independent encoding in inference stage. to this end, the dual-vaes @xcite is proposed by using the question-to-question and answer-to-answer reconstruction as joint training task along with the retrieval task to improve the representation learning, which maintains the independent encoding in inference stage. however, the embeddings produced by dual-encoders or dual-vaes can still only preserve isolated information for questions or answers, while cross information between questions and answers is only learned through similarity score computed by two embeddings. those embeddings preserving isolated semantics can lead to confusing results particularly when an answer can have multiple matched questions and vice versa, which is referred as one-to-many problem @xcite . to address this challenge, @xcite further proposed cross-vaes by reconstructing answers from question embeddings and reconstructing ques-tions from answer embeddings. in such way, the embeddings of questions or answers preserve the cross information from matched answers or questions and improve the performance in one-to-many cases. nevertheless, both dual-vaes and cross-vaes rely on the generation sub-task to enhance the embeddings in retrieval task, while the need of text generation (the word-level joint distribution of sentences) and that of answer retrieval (the sentence-level matching distribution of qa-pairs) are different, which are suspected to conflict in joint training @xcite . it then brings an interesting question: is it feasible to exploit the cross information in retrieval task and keep the independence of sentence encoding in inference stage. in this research we proposed a cross-encoders (details in section 3.3) as an additional guidance during dual-encoders training besides the similarity score. the cross-encoders could form comprehensive representation through cross-attention to reflect the complex relations (e.g., one-to-many) between matched questions and answers. we also developed geometry alignment mechanism (details in section 3.4) as the guiding way to effectively bridge the gap between cross-encoders and dual-encoders by forcing the dual-encoders to mimic cross-encoders on the geometry (i.e., semantic feature structure) in embedding space. the contributions of this paper are in three folds: 1) focusing on the lack of interactions in dual-encoders architecture, we introduce an enhancing dual-encoders with cross-embeddings (endx) framework to solve this limitation, where a cross-encoders model is proposed to guide the training of dual-encoders model; 2) to achieve such enhancement in endx, we propose a novel geometry alignment mechanism (gam) to align the geometry of embeddings from dual-encoders with that from cross-encoders, which models the interactions between words within question and answer. this frees the dual-encoders from having to encode necessary information with no access to matched sentence; 3) to validate our framework, we conduct extensive experiments and show that the proposed framework significantly improves dual-encoders model and outperforms the state-ofart model on multiple qa datasets. ## related work traditional answer retrieval consists of two-stage pipeline including key words matching (bm25 @xcite ) to efficiently retrieve multiple relevant passages and re-ranking by neural network to select correct answers from retrieved results. but it may fall short here as the connection between answers and questions in context is not modelled directly, while the large document where the answer locates could be not highly relevant to the question @xcite . to address the problem in two-stage pipeline retrieval, there is growing interest in training endto-end retrieval systems that can efficiently surface relevant results without an intermediate document retrieval phase @xcite @xcite @xcite . in recent works @xcite @xcite , using dense representation learned by dual-encoders framework outperformed bm25 in large-scale retrieval task. dual-encoders can encode questions and answers independently and thus enables offline processing to support efficient online response, but there exists a bottleneck that impedes the qa alignment for lack of interaction between questions and answers in their independent encoding. ## conclusion in this work, we propose a framework that enhances dual-encoders with cross-embeddings for answer retrieval. a novel geometry alignment mechanism is introduced to align the geometry of dual-encoders with cross-embeddings. extensive experimental results show that our method significantly improves dual-encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets.