Interactive Text Ranking with B ayesian Optimization: A Case Study on Community QA and Summarization (2020)

Abstract: For many NLP applications, such as question answering and summarization, the goal is to select the best solution from a large space of candidates to meet a particular user’s needs. To address the lack of user or task-specific training data, we propose an interactive text ranking approach that actively selects pairs of candidates, from which the user selects the best. Unlike previous strategies, which attempt to learn a ranking across the whole candidate space, our method uses Bayesian optimization to focus the user’s labeling effort on high quality candidates and integrate prior knowledge to cope better with small data scenarios. We apply our method to community question answering (cQA) and extractive multidocument summarization, finding that it significantly outperforms existing interactive approaches. We also show that the ranking function learned by our method is an effective reward function for reinforcement learning, which improves the state of the art for interactive summarization.

## introduction many text ranking tasks are highly specific to an individual user's topic of interest, which presents a challenge for nlp systems that have not been trained to solve that user's problem. consider ranking summaries or answers to non-factoid questions: a good solution requires understanding the topic and the user's information needs @xcite . we address this by proposing an interactive text ranking approach that efficiently gathers user feedback and combines it with predictions from pretrained, generic models. to minimize the amount of effort the user must expend to train a ranker, we learn from pairwise preference labels, in which the user compares two candidates and labels the best one. pairwise preference labels can often be provided faster than ratings or class labels @xcite @xcite , can be used to rank candidates using learning-to-rank @xcite , preference learning @xcite , or best-worst scaling @xcite , or to train a reinforcement learning (rl) agent to find the optimal solution @xcite . to reduce the number of labels a user must provide, a common solution is active learning @xcite . al learns a model by iteratively acquiring labels: at each iteration, it trains a model on the labels collected so far, then uses an acquisition function to quantify the value of querying the user about a particular pair of candidates. the system then chooses the pairs with the highest values, and instructs the user to label them. the acquisition function implements one of many different strategies to minimize the number of interaction rounds, such as reducing uncertainty @xcite by choosing informative labels that help learn the model more quickly. many active learning strategies, such as the pairwise preference learning method of @xcite , aim to learn a good ranking model for all candidates, for example, by querying the annotator about candidates whose rank is most uncertain. however, we often need to find and rank only a small set of good candidates to present to the user. for instance, in question answering, irrelevant answers should not be shown to the user, so their precise ordering is unimportant and users should not waste time ranking them. therefore, by reducing uncertainty for all candidates, uncertainty-based al strategies may waste labels on sorting poor candidates. here, we propose an interactive method for ranking texts that replaces the standard uncertaintybased acquisition functions with acquisition functions for bayesian optimization (bo) @xcite . in general, bo aims to find the maximum of a function while minimizing the number of queries to an oracle. here, we use bo to maximize a ranking function that maps text documents to scores, treating the user as a noisy oracle. our bo active learning strategy minimizes the number of labels needed to find the best candidate, in contrast to uncertainty-based strategies that attempt to learn the entire ranking function. this makes bo better suited to tasks such as question answering, summarization, or translation, where the aim is to find the best candidate and those with low quality can simply be disregarded rather than ranked precisely. in this paper, we define two bo acquisition functions for interactive text ranking. while our approach is designed to adapt a model to a highly specialized task, generic models can provide hints to help us avoid low-quality candidates. therefore, we learn the ranking function itself using a bayesian approach, which integrates prior predictions from a generic model that is not tailored to the user. previous interactive text ranking methods either do not exploit prior information @xcite @xcite , combine heuristics with user feedback after active learning is complete @xcite , or require expensive re-training of a non-bayesian method @xcite . here, we show how bo can use prior information to expedite interactive text ranking. the interactive learning process is shown in algorithm 1 and examples of our system outputs are shown in figures 1 and 2 . our contributions are (1) a bayesian optimization methodology for interactive text ranking that integrates prior predictions with user feedback, (2) acquisition functions for bayesian optimization with pairwise labels, and (3) empirical evaluations on community question answering (cqa) and extractive multidocument summarization, which show that our method brings substantial improvements in ranking and summarization performance input: candidate texts x with feature vectors φ(x) 1 initialize ranking model m; 2 set the training data @xmath0; while |d| < max interactions do 3 for each pair of texts (x a , x b ) in x, compute @xmath1, φ(x b ), d, m); 4 set p i to be the set of batch size pairs with the highest values of v; 5 obtain labels y i from user for pairs p i ; 6 add y i and p i to d ; 7 train model m on the training set d ; end output: return the trained model m and/or its final ranked list of candidate texts in x. algorithm 1: interactive text ranking process with preference learning. (e.g., for cqa, an average 25% increase in answer selection accuracy over the next-best method with 10 rounds of user interaction). we release the complete experimental software for future work. 2 related work interactive learning in nlp. previous work has applied active learning to tasks involving ranking or optimising generated text, including summarization (p.v.s and meyer, 2017), visual question answering @xcite , and translation @xcite . for summarization, @xcite , @xcite and @xcite , train reinforcement learners by querying the user directly for rewards, which requires in the order of 10 5 interactions. @xcite dramatically reduce the number of user interactions to the order of to 10 2 by using active learning to learn a reward function for rl, an approach proposed by @xcite . these previous works use uncertainty sampling strategies, which query the user about the candidates with the most uncertain rankings to try to learn all candidates' rankings with a high degree of confidence. we instead propose to find good candidates using an optimization strategy. @xcite carried out a large empirical study of uncertainty sampling for sentence classification, semantic role labeling and named entity recognition, finding that exploiting model uncertainty estimates provided by bayesian neural networks improved performance. our approach also exploits bayesian uncertainty estimates. bo for preference learning. bayesian approaches using gaussian processes (gps) have previously been used to reduce errors in nlp tasks involving sparse or noisy labels @xcite , making them well-suited to learning from user feedback. gaussian process preference learning (gppl) @xcite enables gp inference with pairwise preference labels. @xcite introduced scalable inference for gppl using stochastic variational inference (svi) @xcite , which outperformed svm and lstm methods at ranking arguments by convincingness. they included a study on active learning with pairwise labels, but tested gppl only with uncertainty sampling, not bo. here, we adapt gppl to summarization and cqa, show how to integrate prior predictions, and propose a bo framework for gppl that facilitates interactive text ranking. @xcite proposed a bo approach for pairwise comparisons but applied the approach only to a material design use case with a very simple feature space. @xcite proposed alternative bo strategies for pairwise preferences, but their approach requires expensive sampling to estimate the utilities, which is too slow for an interactive setting. @xcite also propose bo with pairwise preferences, but again, inference is expensive, the method is only tested with fewer than ten features, and it uses an inferior probability of improvement strategy (see @xcite . our gppl-based framework permits much faster inference even when the input vector has more than 200 features, and hence allows rapid selection of new pairs when querying users. @xcite use bo to select training data for transfer learning in nlp tasks such as sentiment analysis, pos tagging, and parsing. however, unlike our interactive text ranking approach, their work does not involve pairwise comparisons and is not interactive, as the optimizer learns by training and evaluating a model on the selected data. in summary, previous work has not yet devised bo strategies for gppl or suitable alternatives for interactive text ranking. ## background on preference learning popular preference learning models assume that users choose a candidate from a pair with probability p, where p is a function of the candidates' utilities @xcite . utility is defined as the value of a candidate to the user, that is, it quantifies how well that instance meets their needs. when candidates have similar utilities, the user's choice is close to random, while pairs with very different utilities are labeled consistently. such models include the bradley-terry model (bt) @xcite @xcite , and the thurstone-mosteller model @xcite . bt defines the probability that candidate a is preferred to candidate b as follows: to learn the weights, we treat each pairwise label as two data points: the first point has input features @xmath2, and the second point is the reverse pair, with @xmath3. then, we use standard techniques for logistic regression to find the weights w that minimize the l2-regularized cross entropy loss. the resulting linear model can be used to predict labels for any unseen pairs, or to estimate candidate utilities, f @xmath4, which can be used for ranking. uncertainty (unc). at each active learning iteration, the learner requests training labels for candidates that maximize the acquisition function. p.v.s and meyer (2017) proposed an uncertainty sampling acquisition function for interactive document summarization, which defines the uncertainty about a single candidate's utility, u, as follows: u(a|d) =    p(a|d) if p(a|d) ≤ 0.5 1-p(a|d) if p(a|d) > 0.5, unc is intended to focus labeling effort on candidates whose utilities are uncertain. if the learner is uncertain about whether candidate a is a good candidate, p(a|d) will be close to 0.5, so a will have a higher chance of being selected. unfortunately, it is also possible for p(a|d) to be close to 0.5 even if a has been labeled many times if a is a candidate of intermediate utility. therefore, when using unc, labeling effort may be wasted re-labeling mediocre candidates. the problem is that bt cannot distinguish two types of uncertainty. the first is aleatoric uncertainty due to the inherent unpredictability of the phenomenon we wish to model @xcite . for example, when predicting the outcome of a coin toss, we model the outcome as random. similarly, given two equally preferable items, we assume that the user assigns a preference label randomly. it does not matter how much training data we observe: if the items are equally good, we are uncertain which one the user will choose. the second type is epistemic uncertainty due to our lack of knowledge, which can be reduced by acquiring more training data, as this helps us to learn the model's parameters with higher confidence. bt does not quantify aleatoric and epistemic uncertainty separately, unlike bayesian approaches @xcite , so we may repeatedly select items with similar utilities that do not require more labels. to rectify this shortcoming, we replace bt with a bayesian model that both estimates the utility of a candidate and quantifies the epistemic uncertainty in that estimate. ## interactive learning with gppl we now define four acquisition functions for gppl that take advantage of the posterior covariance, c, to account for uncertainty in the utilities. ## experiments we perform experiments on three tasks to test our interactive text ranking approach: (1) community question answering (cqa)-identify the best answer to a given question from a pool of candidate answers; (2) rating extractive multidocument summaries according to a user's preferences; (3) generating an extractive multidocument summary by training a reinforcement learner with the ranking function from 2 as a reward function. using interactive learning to learn the reward function rather than the policy reduces the number of user interactions from many thousands to 100 or less. these tasks involve highly specialized questions or topics where generic models could be improved with user feedback. for the first two tasks, we simulate the interactive process in algorithm 1. the final task uses the results of this process. datasets. both the cqa and multidocument summarization datasets were chosen because the answers and candidate summaries in these datasets are multisentence documents that take longer for users to read compared to tasks such as factoid question-answering. we expect our methods to have the greatest impact in this type of longanswer scenario by minimizing user interaction time. for cqa, we use datasets consisting of questions posted on stackexchange in the communities apple, cooking, and travel, along with their accepted answers and candidate answers taken from related questions @xcite . each accepted answer was marked by the user who posted the question, so reflects that user's own opinion. dataset statistics are shown in for summarization, we use the duc datasets, foot_1 which contain model summaries written by experts for collections of documents related to a narrow topic. each topic has three model summaries, each written by a different expert and therefore reflecting different opinions about what constitutes a good summary. compared with single-document summarization, the challenging multidocument case is an ideal testbed for interactive approaches, because the diversity of themes within a collection of documents makes it difficult to identify a single, concise summary suitable for all users. priors and input vectors. we use our interactive approach to improve a set of prior predictions provided by a pretrained method. for cqa, we first choose the previous state-of-the-art for long answer selection, coala @xcite ,which estimates the relevance of answers to a question by extracting aspects (e.g., n-grams or syntactic structures) from the question and answer texts using cnns, then matching and aggregating the aspects. for each topic, we train an instance of coala on the training split given by @xcite , then run the interactive process on the test set, that is, the dataset in next we apply our interactive approach to refine predictions from the current state of the art @xcite , which we refer to as bert-cqa. this method places two dense layers with 100 and 10 hidden units on top of bert @xcite . as inputs to bert, we concatenate the question and candidate answer and pad sequences to 512 tokens (4% qa pairs are over-length and are truncated). the whole model is fine-tuned on the stackexchange training sets, the same as coala. in our simulations, we use the fine-tuned, finallayer [cls] embeddings with 768 dimensions as inputs to bt and gppl for each question-answer pair. as prior predictions for summary ratings we first evaluate reaper, a heuristic evaluation function described by @xcite . we obtain bigram+ feature vectors for candidate summaries by augmenting bag-of-bigram embeddings with additional features proposed by @xcite . the first 200 dimensions of the feature vector have binary values to indicate the presence of each of the 200 most common bigrams in each topic after tokenizing, stemming and applying a stop-list. the last 5 dimensions contain the following: the fraction of the 200 most common bigrams that are present in the document (coverage ratio); the fraction of the 200 most common bigrams that occur more than once in the document (redundancy ratio); document length divided by 100 (length ratio); the sum over all extracted sentences of the reciprocal of the position of the extracted sentence in its source document (extracted sentence position feature); a single bit to indicate if document length exceeds the length limit of 100 tokens. the same features are used for both tasks (2) learning summary ratings and (3) reinforcement learning. we also test prior predictions from a state-ofthe-art summary scoring method, supert @xcite , which uses a variant of bert that has been fine-tuned on news articles to obtain 1024-dimensional contextualized embeddings of a summary. to score a summary, supert extracts a pseudo-reference summary from the source documents, then compares its embedding with that of the test summary. with the supert priors we compare bigram+ feature vectors and the supert embeddings as input to bt and gppl for task (2). interactive methods. as baselines, we test bt as our preference learner with random selection and the unc active learning strategy, and gppl as the learner with random selection. we also combine gppl with the acquisition functions described in section 4, unpa, eig, imp, and tp. for random sampling, we repeat each experiment ten times. ## conclusions we proposed a novel approach to interactive text ranking that uses bayesian optimization (bo) to identify top-ranked texts by acquiring pairwise feedback from a user and applying gaussian process preference learning (gppl). our experiments showed that bo significantly improves the accuracy of answers chosen in a cqa task with small amounts of feedback, and leads to summaries that better match human-generated model summaries when used to learn a reward function for reinforcement learning. of two proposed bayesian optimization strategies, we found that expected improvement (imp) outperforms thompson sampling (tp) if the goal is to optimize the proposed best solution. tp may require a larger number of interactions due to its random sampling step. imp is effective in both cqa and summarization tasks, but has the strongest impact on cqa with only 10 interactions. this may be due to the greater sparsity of candidates in cqa (100 versus 10,000 for summarization), which allows them to be more easily discriminated by the model, given good training examples. further evaluation with real users is required to gauge the quantity of feedback needed in a particular domain. when using high-dimensional bert embeddings as inputs, gppl requires more labels to achieve substantial improvements. future work may therefore investigate recent dimensionality reduction methods @xcite . we found that performance improves when including prior predictions as the gppl prior mean but it is unclear how best to estimate confidence in the prior predictions-here we assume equal confidence in all prior predictions. future work could address this by adapting the gppl prior covariance matrix to kick-start bo. the method is also currently limited to a single set of prior predictions: in future we intend to integrate predictions from several models.