Beyond [ CLS ] through Ranking by Generation (2020)

Abstract: Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document’s language model, were very successful in various IR tasks in the past. However, with the advent of modern deep neural networks, attention has shifted to discriminative ranking functions that model the semantic similarity of documents and queries instead. Recently, deep generative models such as GPT2 and BART have been shown to be excellent text generators, but their effectiveness as rankers have not been demonstrated yet. In this work, we revisit the generative framework for information retrieval and show that our generative approaches are as effective as state-of-the-art semantic similarity-based discriminative models for the answer selection task. Additionally, we demonstrate the effectiveness of unlikelihood losses for IR.

## introduction most recent approaches for ranking tasks in information retrieval (ir) such as passage ranking and retrieval of semantically related questions have focused primarily on discriminative methods using neural networks that learn a similarity function to compare questions and candidate answers @xcite @xcite @xcite . on the other hand, classical literature on probabilistic models for ir showed that language modeling, a type of simple generative model, can be effective for document ranking @xcite @xcite . the key idea consists of first training a unique language model lm i for each candidate document d i , then using the likelihood of generating the input query using lm i , denoted by p (q|lm i ), as the ranking score for document d i . recent advances in neural language models (nlms) have led to impressive improvements in the quality of automatically generated text @xcite . however, to the best of our knowledge, there is no existing work on exploring the effectiveness of modern generative models such as gpt2, for complex ranking tasks such as answer selection. in this work, we intend to fill this gap by demonstrating that large pretrained generative models can be very effective rankers. unlike classic lm based approaches for ir that employ separate lms for each document, our proposed method uses a single global lm that applies to all documents. the global pretrained generator is fine-tuned on the task of query generation conditioned on document content as the context. additionally, in order to leverage both positive and negative examples, we propose the use of (1) unlikelihood loss on negative examples and (2) ranking loss on the likelihood of positive and negative examples. at inference time, given an input query, our method scores each candidate document using the likelihood of generating the query given the document, as estimated by our fine-tuned global lm. we focus our experiments on the task of answer selection (a.k.a passage ranking). in this task, given an input question and a set of candidate passages, the goal is to rank the candidate passages so that passages containing the correct answer appear at the top of the ranked list. considerable body of work exists on the use of nns for this task @xcite @xcite @xcite , where the most recent ones use bert-based models that perform discrimination based on the special [cls] token @xcite @xcite . a contemporaneous work by @xcite also proposes a generative approach for the passage ranking task. however, while their approach decides the relevance of a passage by generating a single keyword (e.g. true or false), our method uses the conditional likelihood of generating the question given the passage as a relevance score. we perform extensive experiments using gpt2 @xcite and bart @xcite , which are transformer-based lms @xcite that were pretrained using large volumes of textual data. the lms are fine-tuned on four different passage ranking datasets separately: wikipassageqa, wikiqa, insuranceqa v2, and yahooqa. our experimental results indicate that our generative approaches are as effective as stateof-the-art discriminative-based approaches for answer selection. ## conclusion we have proposed a new generative approach for ir based on large pretrained neural language models, and demonstrated their effectiveness as rankers by providing robust experimental results on four different datasets. additionally, we demonstrated that unlikelihood-based losses are effective for allowing the use of negative examples in generative-based information retrieval. we believe that our approach can also be effectively used for text classification problems, where the score of a class label c is computed as the likelihood of generating the class label c given the document d, p(c|d).