Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation (2024)

Abstract: Customizing LLMs for a specific task involves separating high-quality responses from lower-quality ones. This skill can be developed using supervised fine-tuning with extensive human preference data. However, obtaining a large volume of expert-annotated data is costly for most tasks. In this paper, we explore a novel method to optimize LLMs using ranking metrics. This method trains the model to prioritize the best responses from a pool of candidates created for a particular task. Rather than a traditional full ordering, we advocate for a partial ordering, as achieving consensus on the perfect order of candidate responses can be challenging. Our partial ordering is more robust, less sensitive to noise, and can be achieved with limited human annotations or through heuristic methods. We test our system’s improved response generation ability using benchmark datasets, including textual entailment and multi-document question answering. We conduct ablation studies to understand crucial factors, such as how to gather candidate responses for a specific task, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named RESCUE, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.

## introduction a significant advantage of large language models is their ability to explain their predictions @xcite @xcite @xcite . for example, llms may suggest lab tests to doctors based on patient symptoms @xcite or help financial analysts evaluate risks in their investment portfolios @xcite , providing explanations for each. as llms increasingly assist in decision-making across domains, examining the quality of their explanations becomes 1 our code and models are available at: https://github. com/ekonwang/rrescue . ## ranking llm responses candidate responses for a given prompt x, can be organized into a strict order. openai has employed a team of trained human labelers to rank sets of model outputs from best to worst to train a reward model @xcite . however, this method is quite expensive. we propose two cost-effective approaches to establish a partial ordering (po) of responses. our first method, (po) human prioritization, posits that human responses should take priority over model responses, as they offer valid rationales and accurate labels. (po) label prioritization places responses with correct labels above those with incorrect labels, irrespective of whether they are human or model-generated. this is because rationales resulting in correct labels are more valuable than those leading to incorrect labels. the latter may contain flawed reasoning that misguides their predictions. lastly, (po) human-label hy-brid employs a fine-grained grouping. it places human responses above model responses with correct labels, which are then prioritized over responses with incorrect labels. this hierarchy is designed to motivate the llm to generate rationales comparable to humans' or, at a minimum, to produce rationales that lead to accurate labels. partial orderings (po) of responses offer enhanced flexibility and noise robustness. for example, in developing llama-2, @xcite noted that even human labelers struggle to decide between two similar model responses, with annotations for such responses often hinging on subjective judgement and nuanced details. by utilizing a partial order, we only incorporate the most clearcut pairs of model outputs in the ranking metric, thereby improving the quality of response pairs used in model fine-tuning. for comparison, we examine two full ordering (fo) approaches. (fo) similarity embeds each candidate response into a vector, which are then ranked based on their cosine similarity to the vector representing the human response. the second approach (fo) gpt-3.5-turbo leverages the gpt-3.5-turbo-0613 model to rank candidate responses. we instruct it to prioritize candidates with the same labels as the human response, but allowing it to decide whether this criterion is met. we compare full and partial ordering approaches in §6. ## collecting candidate responses we enrich each example with a set of candidate responses, targeting a mix that includes both accurate and inaccurate predictions, along with explanations that are both sound and flawed. we incorporate human annotations into the mix when available. for tasks with long contexts, we anchor responses in different parts of the context to increase diversity. this enriched dataset is used to train our llm to improve its response generation. next, we outline two strategies for generating candidate responses. ## experiments we have chosen llama-2-7b as our base model for task-specific training. the llama-2 series outperforms other open-source options, such as falcon @xcite , mistral @xcite , vicuna @xcite and mpt (mo-saicml, 2023), on a number of tasks. its 7b variant requires significantly less gpu memory, which is crucial for specific domains without the specialized infrastructure to serve larger models. we use adamw @xcite ) with a learning rate of 2e -5 and a cosine scheduler with a 0.03 warmup rate. our training utilizes fully sharded data parallelism and bf16 mixed precision training, which is generally faster, consumes less memory, and is preferable for large models. our experiments are conducted using 4xa100 gpus, and task-specific training is limited to a single epoch for both supervised fine-tuning and response ranking. this is to mitigate the risk of multi-epoch degradation @xcite and potential overfitting from repeated exposure to the training data. the batch size is set at @xmath0, the same configuration used for llama-2 @xcite . it is calculated as the product of three factors, @xmath1, combining gradient accumulation steps (@xmath2, per-gpu batch size (@xmath3, and the number of gpus (@xmath4. this strategy allows us to handle a large number of candidates during response ranking. ## discussion balancing coefficient our approach uses a hyperparameter α to balance the impact of supervised fine-tuning and the ranking metric. figure 4 shows the influence of different α on task accuracy. we find that optimal performance is achieved with an α value between 0.01 to 0.1. our results indicate that, while supervised fine-tuning is pivotal for res-cue, integrating the ranking metric enhances the method's robustness to noise. ## conclusion in this paper, we introduce rescue, an approach that trains the llm to prioritize sound responses over erroneous ones, thereby enhancing overall task accuracy and the quality of explanations. accurate model predictions often come with high-quality explanations. we build on this insight to rank candidate responses using a partial ordering approach, as achieving consensus on the perfect order of responses is challenging. rescue has demonstrated competitive performance on benchmarks.