Exploring Dual Encoder Architectures for Question Answering (2022)

Abstract: Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. There are two major types of dual encoders, Siamese Dual Encoders (SDE), with parameters shared across two encoders, and Asymmetric Dual Encoder (ADE), with two distinctly parameterized encoders. In this work, we explore the dual encoder architectures for QA retrieval tasks. By evaluating on MS MARCO, open domain NQ, and the MultiReQA benchmarks, we show that SDE performs significantly better than ADE. We further propose three different improved versions of ADEs. Based on the evaluation of QA retrieval tasks and direct analysis of the embeddings, we demonstrate that sharing parameters in projection layers would enable ADEs to perform competitively with SDEs.

## introduction a dual encoder is an architecture consisting of two encoders, each of which encodes an input (such as a piece of text) into an embedding, and where the model is optimized based on similarity metrics in the embedding space. it has shown an excellent performance in a wide range of information retrieval and question answering tasks @xcite . this approach is also easy to productionize because the embedding index of dual encoders can grow dynamically for code and additional information: https://sites.google.com/view/ explore-dual-encoder-architect . ## related work dual encoders have been widely studied in entity linking @xcite , open-domain question answering @xcite , and dense retrieval @xcite , etc. this architecture consists of two encoders, where each encoder encodes arbitrary inputs that may differ in type or granularity, such as queries, images, answers, passages, or documents. open-domain question answering (odqa) is a challenging task that searches for evidence across large-scale corpora and provides answers to user queries @xcite . one of the prevalent paradigms for odqa is a two-step approach, consisting of a retriever to find relevant evidence and a reader to synthesize answers. alternative approaches are directly retrieving from large candidate corpus to provide sentence-level @xcite or phrase-level @xcite answers; or directly generating answers or passage indices using an end-to-end generation approach @xcite . @xcite compared the performance of sdes and ades for phrase-level qa retrieval tasks. however, they only considered the two extreme cases, where two towers have the parameters completely shared or distinct. in this work, we address the missing piece of previous work by exploring parameter sharing in different parts of the model. ## method we follow a standard setup of qa retrieval: given a question q and a corpus of answer candidates a, the goal is to retrieve k relevant answers a @xmath0. the answer can be either a passage, a sentence, or a phrase. we adopt a dual encoder architecture @xcite as the model to match query and answers for retrieval. the model has two encoders, where each is a transformer that encodes a question or an answer. each encoder first produces a fixed-length representation for its input and then applies a projection layer to generate the final embedding. we train the dual encoder model by optimizing the contrastive loss with an in-batch sampled softmax @xcite : ## experiments and analysis we evaluate the proposed dual encoder architectures on six question-answering retrieval tasks from ms marco @xcite and multireqa @xcite . in ms marco, we consider the relevant passages as answer candidates, while for the five qa datasets in multireqa the answer candidates are individual sentences. we further validate the conclusion on an open domain questionanswering task, open domain naturalquestions, where the retrieval candidates are context passages. to initialize the parameters of dual encoders, we use pre-trained t5.1.1 encoders @xcite . following ni et al. (2021b) , we take the average embeddings of the t5 encoder's outputs and send to a projection layer to get the final embeddings. the projection layers are randomly initialized, with variance scaling initialization with scale 1.0. for the retrieval, we use mean embeddings from the encoder towers. to make a fair comparison, the same hyper-parameters are applied across all the models for the fine-tuning with adafactor optimizer @xcite , using learning rate 10 -3 and batch size 512. the models are fine-tuned for 20, 000 steps, with linear decay of learning rate from 10 -3 to 0 at the final steps. the fine-tuned models are benchmarked with precision at 1 (p@1) and mean reciprocal rank (mrr) on the qa retrieval tasks, in ## conclusion and future work based on the experiments on 7 qa retrieval tasks with three different model sizes, we conclude that, although sdes outperforms ades, sharing the pro-jection layer between the two encoders enables ades to perform competitively with sdes. by directly probing the embedding space, we demonstrate that the shared projection layers in sde and ade-spl maps the embeddings of the two encoder towers into coinciding parameter spaces, which is crucial for improving the retrieval quality. therefore, we recommend to share the projection layers between two encoders of ades in practice.