Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets (2024)

Abstract: Training Large Language Models (LLMs) with Reinforcement Learning from AI Feedback (RLAIF) aligns model outputs more closely with human preferences. This involves an evaluator model ranking multiple candidate responses to user prompts. However, the rankings from popular evaluator models such as GPT-4 can be inconsistent.We propose the Repeat Ranking method, in which we evaluate the same responses multiple times and train only on those responses which are consistently ranked. Using 2,714 training prompts in 62 languages, we generated responses from 7 top multilingual LLMs and had GPT-4 rank them five times each. Evaluating on MT-Bench chat benchmarks in six languages, our method outperformed the standard practice of training on all available prompts.Our work highlights the quality versus quantity trade-off in RLAIF dataset generation and offers a stackable strategy for enhancing dataset and thus model quality.

## introduction reinforcement learning has been shown to improve large language model (llm) performance significantly @xcite , with this form of learning instructing an llm both how to and how not to generate text. this has come in the forms of reinforcement learning from human feedback (rlhf) @xcite and reinforcement learning from artificial intelligence feedback (rlaif) @xcite , where a human or ai is used, respectively, to determine the relative quality of multiple responses to a given prompt. based on these quality rankings, high quality and low quality responses are defined as "positive" and "negative" and this preference dataset is then used to train an llm either with the help of a reward model or by directly training using a method such as proximal policy optimisation (ppo) @xcite , direct policy optimisation (dpo) @xcite , or odds ratio preference optimisation (orpo) @xcite . this style of training has lead to many of the improvements in recent years in llm training, with both gpt-3.5 @xcite , trained with rlhf, and starling @xcite , trained with rlaif, demonstrating gains upon previous state-of-the-art performance across many evaluation benchmarks. most publicly available preference data is monolingual, but we hypothesize that training a model on multilingual preference data will improve the resultant model's multilingual capabilities. this prompted us to create a multilingual preference dataset. we follow previous methods for creating hlaif preference datasets such as nectar @xcite by first sampling human generated prompts from public datasets before generating various responses to each prompt using seven state-of-the-art llms. we then use a state-of-the-art llm, gpt-4, to evaluate the relative ranking of each response. however, we found that when the evaluation process was repeated on the same responses, different rankings were sometimes output by gpt-4. this suggested that the definition of positive and negative labels in these instances had a lower confidence than instances where gpt-4 would consistently output the same ranking given a set of responses. therefore, we hypothesized that training only on rankings that gpt-4 consistently outputs over multiple evaluations would lead to greater downstream evaluation performance compared to training on all rankings, both consistent and inconsistent. this lead us to propose the repeat ranking method, whereby responses are evaluated multiple times and the consistency of the rankings is used as a filter for inclusion or exclusion from the training set. a representation of our repeated ranking method can be found in fig. 1 . prompt 1: what is the capital of france? model a: paris -the city of love model b: the capital of france is paris. model c: paris is the capital of france. prompt 2: what was the world's population in 2023? model a: 8 thousand model b: 8 billion model c: 5 billion evaluation 1.1: @xmath0.2: @xmath1.3: @xmath2.1: @xmath3.2: @xmath4.3: @xmath5: paris model d: 1 thousand kendall's w of evaluation @xmath6.111 result: exclude from training data kendall's w of evaluation @xmath7.644 result: use model b as positive response and model d as negative response in training data figure 1 : a visual description of how we select our data for training. we use our repeat ranking method to repeat the evaluations of the models multiple times and then only train on the best and worst responses which have a high kendall's w, a measure of ranking agreement, associated with their ranking. we conducted experiments training an llm using all rankings, as well as the 75%, 50%, and 25% most consistent rankings. we then evaluated each trained model using the mt-bench benchmark across 6 languages. our results show that training on the more consistently ranked responses gives greater downstream evaluation performance compared to training on all data for a majority of languages tested. our findings inform the creation of future preference datasets and offer a method of improving the quality of existing preference datasets. this may open up exciting new avenues for training llms and highlights the importance of high quality positive and negative data when training using rlaif. we make our training data 1 , training code 2 , and 1 https://huggingface.co/datasets/lightblue/ mitsu 2 https://github.com/lightblue-tech/suzume/ tree/main/mitsu trained models foot_0 available online. ## related work llm chat performance has been improved by training on rlhf datasets in multiple works within the literature. the rlhf dataset used to train instructgpt was created by having users and paid annotators evaluate multiple responses to a given prompt and indicating their preferred prompt @xcite . this work stated that "most comparisons are only labeled by 1 contractor for cost reasons" and that "having examples labeled multiple times could help identify areas where our contractors disagree, and thus where a single model is unlikely to align to all of them", indicating the seeming importance of having consistently similarly ranked preference data when training with rlhf. in contrast, the openassistant conversations (oasst1) dataset @xcite , contains conversation prompts and responses that are written by volunteers, with the responses evaluated by multiple volunteers. while this is a large dataset of more than 10,000 individual messages, over 70% of these conversations are in either english or spanish, reducing oasst1's applicability to training a multilingual model. generating data using human labellers is also costly, which is why several datasets have been constructed for rlaif. previous work includes the use of "constitutional ai" @xcite ) whereby an llm is prompted to respond to a prompt before being tasked with revising that response to be less harmful and in line with principles set by researchers. the llm then generates a less harmful response and the original and revised responses are then used to train another llm using reinforcement learning. further work showed that training using rlaif can lead to similar human evaluation scores compared to rlhf @xcite . this work also showed that rlaif by training directly on response evaluation scores elicited from llms achieves greater down-stream task performance compared to the constitutional ai approach of having an llm revise existing responses. nectar @xcite is a preference dataset which first samples prompts from a variety of open source datasets, before generating responses based on these prompts using seven state-of-theart llms (gpt-4, gpt-3.5-turbo, gpt-3.5-turboinstruct, command r+, command r, llama-2-7bchat, and mistral-7b-instruct). these responses are then ranked once by gpt-4 and these rankings are used to train the starling alpha and beta models using reinforcement learning. these prompts and responses are also all in english, meaning that this dataset is not suitable for training a multilingual model. due to the paucity of high quality multilingual models existing within the literature, we create one, which we call mitsu. previous work has also shown that filtering reinforcement learning data can lead to higher downstream task accuracy @xcite . however, this approach relies on an external reward model to choose which data to filter, limiting the application of this approach to domains and languages that no existing reward model has been trained on. ## method the overall objective of this piece of work was to create an llm that was more proficient at multilingual chat than previous llms. in the course of creating such an llm, we generated also insights into the process of creating high quality preference datasets. this section details how we used our repeated ranking method to make our training dataset named mitsu, how we trained our model, and finally how we evaluated our llm. ## results while the base model did not exceed gpt-3.5-turbo in any language, suzume-orpo-50 outperformed gpt-3.5-turbo in 4 out of 6 languages, demonstrating that orpo training enables llms match or surpass gpt-3.5-turbo on chat benchmarks. however, gpt-3.5-turbo still led in english and japanese. we also conducted other small scale tests to fur-ther probe the effects of orpo training. one notable test (suzume-orpo-gpt on another test (llama-orpo-50 on the final small scale test (suzume-orporandom-50 on the belebele scores for each of our trained models can be found in ## discussion our results demonstrate the importance of orpo training in improving the chat abilities of finetuned models. this, in turn, highlights the importance of creating high quality preference datasets to train llms using the orpo method. our results showing that model trained on less, but more consistently evaluated, preferences can achieve greater chat benchmark performance than training on all the data. this has the double benefits of increasing performance while reducing training cost by as much as four times for training on our 25% training subset. however, the extra inference computation required to rank responses multiple times is an increased cost with this method of dataset creation. this could benefit both current and future datasets, with datasets such as nectar @xcite potentially being improved by re-evaluating the dataset's responses and filtering out less consis-tently evaluated rows. we theorize that the correct balance between consistency and data volume (i.e. where the cut-off for kendall's w would be) may vary between tasks, but we have shown that for our multilingual chat setting the benefit on evaluation performance of having a threshold above which we keep our data. our results are also purely dataset-based, meaning that they might be able to be stacked with other recent llm training methods such as simpo @xcite and expo @xcite . ## future work our results suggest that the technique of repeated evaluations on preference data and only keeping the consistently evaluated prompts and responses for training could be applied to other rlaif and rlhf datasets. future work could include investigating whether training only using prompts and responses with high agreement in the evaluations from human annotators could lead to higher accuracy than training on all prompts and responses. another potential avenue for future work is using more than one evaluator model for ranking responses. in this work, we only used gpt-4, but there are other state-of-the-art llms such as claude 3 (anthropic, 2024) and gemini 1.5 pro @xcite . we theorize that combining the evaluations of multiple high performance llms could serve to create more robust evaluations of responses and mitigate the demonstrated bias that any one llm exhibits @xcite . the mitsu dataset that we use to train our model is single-turn, meaning that each example the mitsu dataset also consists of prompts sampled from the tagengo dataset @xcite , which are derived from users prompts to llms hosted on a demo site. we theorize that these prompts are a mixture of easy and difficult for an llm to answer. training on tasks that llms are already highly proficient at might be a waste of training resources, so future work could filter prompts based on their perceived difficulty for llms. we believe that this may improve llms abilities on these difficult tasks. in our experiments, we chose to rank responses 5 times due to that being the financial limit of our experiment. however, future work could empirically find an optimal number of times to repeat evaluations to obtain a reliable kendall's w score. a slight limitation of the repeated ranking approach is the increased inference cost in evaluating responses multiple times as an analogue for determining the confidence of the ranking model in the ranking. future work could explore mitigating this effect by evaluating the combined log probability of a single ranking output and training using only the responses from rankings with the highest probability. tools and agents have also been shown to augment the abilities of llms @xcite @xcite . future work could explore using tools or agents to enhance the evaluation abilities of the evaluator llm when evaluating prompt responses. for example, a search tool could determine the veracity of factual claims, or a calculator tool would be able to confirm the mathematical results of an llm. we theorize that this would lead to more accurate evaluation and would ultimately lead to more accurate llms. ## conclusion in this study, we explored the impact of repeated rankings from an ai evaluator (gpt-4) on training reinforcement learning from ai feedback (rlaif) models for multilingual chat capabilities. we found that responses evaluated consistently by gpt-4 led to higher downstream performance across multiple languages, compared to training on all data regardless of evaluation consistency. our findings indicate that selective training based on evaluation consistency can enhance chat performance and offer a method to improve existing preference datasets. this highlights the balance between quality and quantity when constructing datasets for rlaif. our work opens avenues for further optimizing rlaif datasets and refining training methodologies to develop more proficient multilingual llms.