Tuna: Instruction Tuning using Feedback from Large Language Models (2023)

Abstract: Instruction tuning of open-source large language models (LLMs) like LLaMA, using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4, has proven to be a cost-effective way to align model behaviors with human preferences. However, the instruction-tuned model has only seen one response per instruction, lacking the knowledge of potentially better responses. In this paper, we propose finetuning an instruction-tuned LLM using our novel probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs. Furthermore, we apply probabilistic ranking and contextual ranking sequentially to the instruction-tuned LLM. The resulting model, which we call Tuna, consistently improves the performance on Super Natural Instructions (119 test tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results than several strong reinforcement learning baselines. Our code and data are available at https://github.com/microsoft/LMOps.

## introduction large language models (llms) have made significant progress by scaling up model size and data size @xcite @xcite @xcite for unsupervised pre-training and subsequently applying reinforcement learning from human feedback (rlhf) to align model responses with human preferences @xcite . more recently, instruction tuning @xcite , 2022) with self-instruct algorithm @xcite has emerged as a cost-effective method for aligning with human preferences. in this approach, open llms like llama @xcite can be finetuned on instruction-following data generated by openai gpt using the self-instruct algorithm. the alpaca model @xcite exemplifies this technique, which enables close alignment with human preferences while reducing dependence on human-labeled data. however, instruction tuning offers only a broad guideline for the base llms to transition from "next token prediction" to a more interactive, instruction-following style. as a result, the model may learn some superficial features or styles from the instruction data but still lacks a deeper understanding of what constitutes a preferred response. for instance, when given a question like "give three tips for staying healthy", a base llm may generate fluent yet undesirable continuations, while an instruction-tuned llm could offer three general tips. humans might prefer more detailed tips over general tips, but such tips are less likely to be sampled since they have lower likelihood within the current model distribution. this can be attributed to the fact that they are either unseen during instruction tuning or hard to be sampled due to the exposure bias @xcite . to address this, we propose further finetuning of an instruction-tuned llm to discern the quality of multiple responses more precisely, using our novel probabilistic ranking (sec. 2.2; fig. 1 top ) and contextual ranking (sec. 2.3; fig. 1 middle) approaches. probabilistic ranking enables the instruction-tuned llm to inherit the high-quality and low-quality responses as well as their relative rankings from the teacher llm (e.g., textdavinci-003). in contrast, contextual ranking aims to re-balance the instruction-tuned model's own response distribution with the help of stronger llms (e.g., gpt-4), mitigating the exposure bias issue. we apply probabilistic ranking and contextual ranking sequentially to an instruction-tuned model, i.e., alpaca @xcite , resulting in a model called tuna (sec. 2.4; fig. 1 bottom ). we evaluate tuna on various benchmarks, including super natural instructions @xcite , which contains 119 diverse test tasks; lmentry @xcite , comprising 25 tasks to assess the basic capabilities and robustness of llms; and vicuna qa @xcite which evaluates the model's ability to answer a diverse set of questions with the assistance of gpt-4. experimental results demonstrate that the tuna model not only consistently outperforms the standard instruction-tuned models on all benchmarks, but also surpasses several strong rlhf baselines @xcite . to summarize, our contributions are as follows: ## methodology in this section, we describe how to obtain our tuna model using the feedback from llms. we first describe the vanilla instruction tuning. we then introduce our probabilistic ranking and contextual ranking approaches. lastly, we describe how to integrate both ranking approaches. ## related work instruction tuning instruction tuning aims to improve the usability of base language models @xcite @xcite by finetuning them on instructionresponse pairs in a zero-shot @xcite or few-shot manner @xcite @xcite . the instruction data can be sourced from off-the-shelf nlp benchmarks @xcite @xcite or generated by llms @xcite @xcite . ranking loss learning through re-ranking sequence-level outputs has been studied in sequence-to-sequence models @xcite @xcite . brio and moca algorithms @xcite adopt a pairwise ranking loss to guide the model to generate summaries with higher rouge scores @xcite . in this paper, we use gpt-4's (openai, 2023) strong contextual understanding ability and text-davinci-003's @xcite intrinsic probability measures for ranking. in parallel with our work, @xcite also propose pairwise ranking loss for finetuning llms. key differences include: 1) our pipeline finetuning strategy; 2) our focus on ranking the model's responses; 3) our use of the original response for cross entropy regularization, while they select the highest-reward response. additionally, @xcite also employs gpt models for finetuning bart @xcite on the summarization task. pre-trained model evaluation large pretrained models are powerful evaluation metrics due to their strong contextual understanding ability, such as bertscore @xcite , bartscore @xcite , moverscore @xcite , comet @xcite , and gptscore @xcite . more recently, there are more evaluation strategies based on gpt-3.5 and gpt-4 @xcite . ## conclusion in this paper, we propose to finetune an instructiontuned llm using our probabilistic ranking approach (tuna p ), contextual ranking approach (tuna c ), and a combination of both (tuna). our comprehensive experiments demonstrate consistent performance improvements across three benchmarks: super natural instructions (119 test tasks), lmentry (25 test tasks), and vicuna qa. furthermore, our methods outperform popular reinforcement learning from human feedback baselines that rely on the proximal policy optimization algorithm. these findings underscore the effectiveness of our approach in enhancing the performance of instruction-tuned llms and pave the way for future research in this area.