Pre-Training Methods for Question Reranking (2024)

Abstract: One interesting approach to Question Answering (QA) is to search for semantically similar questions, which have been answered before. This task is different from answer retrieval as it focuses on questions rather than only on the answers, therefore it requires different model training on different data.In this work, we introduce a novel unsupervised pre-training method specialized for retrieving and ranking questions. This leverages (i) knowledge distillation from a basic question retrieval model, and (ii) new pre-training task and objective for learning to rank questions in terms of their relevance with the query. Our experiments show that (i) the proposed technique achieves state-of-the-art performance on QRC and Quora-match datasets, and (ii) the benefit of combining re-ranking and retrieval models.

## introduction an effective approach for answering user questions is to find semantically identical questions, which have been previously answered. although this method cannot be applied to completely new questions, it provides optimal solutions for applications such as frequently asked questions (faqs) @xcite , forum services @xcite , and qa caching systems @xcite , as it provides cheaper and more efficient access to answers than the system generated them. these data base-based qa systems (dbqas) typically consist of three components: (i) a db of questions with their answers, (ii) a retrieval model, which given a question, q, retrieves its most similar questions, and (iii) a selection model, which can more accurately rerank the questions in terms of semantically equivalence. the answer associated with the top-ranked question is typically used as the system output. the fine-tuning of the retrieval * work done as an intern at amazon and ranking models requires training data, labeled in a ranking fashion, i.e., given the query (target question), its top similar k questions needs to be labelled as semantically equivalent or not. while datasets, e.g., quora, constituted by annotated samples of question-question pairs can be used for an initial training, ranking data is essential to obtain optimal accuracy. unfortunately, these datasets require intensive and costly annotation processes and resources to be built. for example, even an annotation workflow built using amazon mechanical turk, is costly foot_0 . alternative approaches to reducing the amount of data have been proposed, ranging from data augmentation @xcite to specialized pre-training (pt) techniques that are aligned with the downstream task. for example, @xcite proposed the the inverse cloze task, an unsupervised pt technique based on a discriminative objective that captures some features of answer retrieval @xcite pretrained on wikipedia, simulating the task of answer sentence selection (as2), by selecting sentences that belong or not to the same document or paragraphs. these methods focus on the relation between question and answer pairs, rather than between two questions, and, most importantly, they do not model the ranking task. in this work, we propose a novel pt technique using a loss function and a data, which surrogate a question re-ranking task. we generate an unsupervised dataset consisting of 18m examples using a re-implementation of the qadbs proposed by @xcite , where each example comprises a question and a rank of five question-answer pairs. to generate pt data, we then swap the first qa pair with another one. the pt task consists in detecting whether the order of qa pairs in the rank is correct or it has been modified. this innovative approach both exploits (i) a new loss function and (ii) knowledge distilled from the retrieval model, i.e., the initial rank. we tested our pt techniques for question re-ranking on two different datasets: (i) qrc @xcite , a question ranking resource designed for dbqas training and testing, and (ii) quora-match @xcite , a binaryclassification over question pairs. the results show that our approach achieves state-of-the-art performance on these benchmarks, e.g., +2% in question selection accuracy on qrc. moreover, we show interesting synergies between re-ranking pt and existing retrieval models, which can be further explored. ## related work various pt techniques have been developed for transformer-based architectures. most of them are based on general and intuitive tasks that can be applied over plain texts. these tasks are designed to teach the model to extract actionable information from text and to learn semantic patterns. first and foremost, masked language model (mlm) pt task was introduced in bert @xcite , where the objective consists of predicting a small fraction of masked tokens, the same pt was applied to various other models, including roberta @xcite and minilm @xcite , showing remarkable results in various downstream applications, including qa and semantic text similarity. alternative pt techniques were proposed by changing the mlm objective: (i) permutation language model (plm) @xcite , where the model tries to predict the next token (left-to-right) of a sentence, whose tokens were permuted; (ii) random token detection (rtd), where the model is trained to find a small amount of tokens replaced with plausible alternatives, generated by a separate model (electra by @xcite ); (iii) random token swap (rts) @xcite , similarly to rtd, the model discriminates the original tokens from those swapped with tokens from the vocabulary; and (iv) text-to-text objective @xcite , where spans of texts are masked to train the model generating coherent sequences. (v) @xcite replace tokens according to text normalization substitutions. finally, (vi) @xcite improves the way electra select complex tokens in rtd. all the above techniques target individual tokens with operations, masking, swapping, replacing them. in contrast, our approach model the entire questions, requiring their classification in the objective function. a closer work to ours are sentence-based techniques, which take multiple sentences as input and try to categorize them: (i) next sentence prediction (nsp) @xcite tries to predict if two input sentences appear side by side in a text or not. (ii) declutr @xcite uses a contrastive learning objective to predict if two sentences come from the same document. ( iii ## question ranking pre-training we create pre-training data using (i) a basic qadbs to generate query/question rank data, and (ii) modifying the rank to simulate the ranking objective. qadbs: this consists of (i) a db of 38m q/a pairs, including 6m q/a pairs from @xcite and 32m additional pairs from paq @xcite ; (ii) a dense retrieval architecture of 33m parameters we built on top of minilm-12l-v2 (pre-trained on a corpus of 900 million sentence pairs for semantic text similarity @xcite ). we fine-tuned it using qrc (see details on appendix a). the retrieval model is a sentence-encoder, which generates the query embedding and, then, computes the cosine-similarity with the pre-computed embeddings associated with each q/a pair stored in the db. this means that it can efficiently sort the entire db, and returns the top k q/a pairs. ## experiments we compared our pt approach with several baselines on qrc and quora datasets. ## conclusion we introduced a novel pt technique to improve models for question ranking tasks. this consists in distilling knowledge from a question retrieval model through unsupervised data generation. our experiments show a clear improvement on two different benchmarks. we share our code, generated data, and models 3 to support future research on this topic.