The Fellowship of the LLM s: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation (2025)

Abstract: This paper presents a novel methodology for generating synthetic Preference Optimization (PO) datasets using multi-model workflows. We evaluate the effectiveness and potential of these workflows in automating and enhancing the dataset generation process. PO dataset generation requires two modules: (1) \textit{response evaluation} , and (2) \textit{response generation} . In the \textit{response evaluation} module, the responses from Large Language Models (LLMs) are evaluated and ranked - a task typically carried out by human annotators that we automate using LLMs. We assess the response evaluation module in a 2 step process. In step 1, we assess LLMs as evaluators using three distinct prompting strategies. In step 2, we apply the winning prompting strategy to compare the performance of LLM-as-a-Judge, LLMs-as-a-Jury, and LLM Debate. Our evaluation shows that GPT-4o-as-a-Judge is more consistent across all datasets. For the \textit{response generation} module, we use the identified LLM evaluator configuration and compare different configurations of the LLM Feedback Loop. We use the win rate to determine the best multi-model configuration for generation. Experimenting with various configurations, we find that the LLM Feedback Loop, with Llama as the generator and Gemma as the reviewer, achieves a notable 71.8% and 73.8% win rate over single-model Llama and Gemma, respectively. After identifying the best configurations for both modules, we generate our PO datasets using the above pipeline.

## introduction large language models (llms) demonstrate a range of natural language processing (nlp) capabilities, including text generation, question answering, and language understanding. however, llms can sometimes deviate from user instructions and exhibit unintended behaviors @xcite . * these authors contributed equally to this work. ## results and discussion the direct comparison strategy performs better than the independent scoring approach across most llms, with a notable improvement for gpt-4o (0.372 vs. 0.249) and gpt-4o-mini (0.342 vs. 0.254). however, it generally falls short when compared to the combined scoring method, where gpt-4o achieves a score of 0.401 using the scoringout-of-100 scale. the higher cohen's kappa values indicate that the direct comparison and combined scoring strategy benefits from providing the llm with a side-by-side evaluation of responses, allowing for more accurate and consistent judgments. the combined scoring strategy, as presented in llm-as-a-judge. the llm-as-judge evaluations, as shown in ## conclusion this paper presents po datasets generated using multi-model frameworks, and evaluates these frameworks by highlighting the advantages, drawbacks, and challenges of each approach. in the response evaluation module, our comparative analysis of llm-as-a-judge, llms-as-a-jury, and llm debate shows the suitability of each setup depending on the context of use. for the response generation module, we evaluate the llm feedback loop using llama-3.1-8b and gemma-2-9b in various configurations. llm-as-a-judge proved to be highly effective when candidate responses don't have a response from the judge llm. whereas llms-as-a-jury and llm debate demonstrated robustness, particularly useful in reducing evaluator bias. however, cohen's kappa for both of these approaches has a high variance making them less suitable for novel applications. our experiments with llm feedback loop using llama-3.1-8b and gemma-2-9b configurations show the potential of multi-model frameworks in refined content generation. configurations where llama-3.1-8b served as the generator and gemma-2-9b as the reviewer consistently delivered better results, demonstrating the benefits of leveraging complementary strengths of different models to refine output quality. these findings indicate the effectiveness of multi-model frameworks for varied ai applications, showing promise for moving towards systems requiring minimal human intervention -however, this method is computationally expensive in comparison. we also generate multiple dpo and kpo datasets using llm feedback loop with llama-3.1-8b as the generator and gemma-2-9b as the evaluator and gpt-4o-as-a-judge. the aim of these datasets is to improve single-model capabilities for better response generation and multi-model capabilities including better communication and improved feedback.