Crossing Variational Autoencoders for Answer Retrieval (2020)

Abstract: Answer retrieval is to find the most aligned answer from a large set of candidates given a question. Learning vector representations of questions/answers is the key factor. Question-answer alignment and question/answer semantics are two important signals for learning the representations. Existing methods learned semantic representations with dual encoders or dual variational auto-encoders. The semantic information was learned from language models or question-to-question (answer-to-answer) generative processes. However, the alignment and semantics were too separate to capture the aligned semantics between question and answer. In this work, we propose to cross variational auto-encoders by generating questions with aligned answers and generating answers with aligned questions. Experiments show that our method outperforms the state-of-the-art answer retrieval method on SQuAD.

## introduction answer retrieval is to find the most aligned answer from a large set of candidates given a question @xcite . it has been paid increasing attention by the nlp and information retrieval community @xcite . sentence-level answer retrieval approaches rely on learning vector representations (i.e., embeddings) of questions and answers from pairs of questionanswer texts. the question-answer alignment and question/answer semantics are expected to be preserved in the representations. in other words, the question/answer embeddings must reflect their semantics in the texts of being aligned as pairs. one popular scheme "dual-encoders" (also known as "siamese network" @xcite ) has two separate encoders to generate question and answer embeddings and question (1): what three stadiums did the nfl decide between for the game? question (2): what three cities did the nfl consider for the game of super bowl 50? a predictor to match two embedding vectors @xcite . unfortunately, it has been shown difficult to train deep encoders with the weak signal of matching prediction @xcite . then there has been growing interests in developing deep generative models such as variational auto-encoders (vaes) and generative adversial networks (gans) for learning text embeddings @xcite . as shown in figure 1(b) , the scheme of "dual-vaes" has two vaes, one for question and the other for answer @xcite . it used the tasks of generating reasonable question and answer texts from latent spaces for preserving semantics into the latent representations. although dual-vaes was trained jointly on question-to-question and answer-to-answer reconstruction, the question and answer embeddings can only preserve isolated semantics of themselves. in the model, the q-a alignment and q/a semantics were too separate to capture the aligned semantics (as we mentioned at the end of the first paragraph) between question and answer. learning the alignment with the weak q-a matching signal, though now based on generatable embeddings, can lead to confusing results, when (1) different questions have similar answers and (2) similar questions have different answers. our idea is that if aligned semantics were preserved, the embeddings of a question would be able to generate its answer, and the embeddings of an answer would be able to generate the corresponding question. in this work, we propose to cross variational auto-encoders, shown in figure 1(c), by reconstructing answers from question embeddings and reconstructing questions from answer embeddings. note that compared with dual-vaes, the encoders do not change but decoders work across the question and answer semantics. experiments show that our method improves mrr and r@1 over the state-of-the-art method by 1.06% and 2.44% on squad, respectively. on a subset of the data where any answer has at least 10 different aligned questions, our method improves mrr and r@1 by 1.46% and 3.65%, respectively. ## related work answer retrieval (ar) is defined as the answer of a candidate question is obtained by finding the most similar answer between multiple candidate answers @xcite . while another popular task on squad dataset is machine reading comprehension (mrc), which is introduced to ask the machine to answer questions based on one given context @xcite . in this section, we review existing work related to answer retrieval and variational autoencoders. answer retrieval. it has been widely studied with information retrieval techniques and has received increasing attention in the recent years by considering deep neural network approaches. recent works have proposed different deep neural models in text-based qa which compares two segments of texts and produces a similarity score. document-level retrieval @xcite @xcite has been studied on many public datasets including including squad @xcite , msmarco @xcite and nq @xcite etc. reqa proposed to investigate sentence-level retrieval and provided strong baselines over a reproducible construction of a retrieval evaluation set from the squad data @xcite . we also focus on sentence-level answer retrieval. variational autoencoders. vae consists of encoder and generator networks which encode a data example to a latent representation and generate samples from the latent space, respectively (kingma and welling, 2013). recent advances in neural variational inference have manifested deep latent-variable models for natural language processing tasks @xcite @xcite . the general idea is to map the sentence into a continuous latent variable, or code, via an inference network (encoder), and then use the generative network (decoder) to reconstruct the input sentence conditioned on samples from the latent code (via its posterior distribution). recent work in cross-modal generation adopted cross alignment vaes to jointly learn rep-resentative features from multiple modalities @xcite @xcite . deconv-lvm @xcite and var-siamese @xcite are most relevant to us, both of which adopt dual-vaes models (see figure 1(b) ) for two text sequence matching task. in our work, we propose a cross-vaes for questions and answers alignment to enhance qa matching performance. ## proposed method problem definition. suppose we have a question set q and an answer set a. each question and answer have only one sentence. each question @xmath0, a, y), where y is a binary variable indicating whether q and a are aligned. therefore, the solution of sentence-level retrieval task could be considered as a matching problem. given a question q and a list of answer candidates c(q) ⊂ a, our goal is to predict p(y|q, a) of each input question q with each answer candidate @xmath1. ## conclusion given a candidate question, answer retrieval aims to find the most similar answer text between candidate answer texts. in this paper, we proposed to cross variational autoencoders by generating questions with aligned answers and generating answers with aligned questions. experiments show that our method improves mrr and r@1 over the best baseline by 1.06% and 2.44% on squad.