Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LM s (2023)

Abstract: Using in-context learning (ICL) for data generation, techniques such as Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023) can train strong conversational agents with only a small amount of human supervision. One limitation of these approaches is that they resort to very large language models (around 175B parameters) that are also proprietary and non-public. Here we explore the application of such techniques to language models that are much smaller (around 10B–40B parameters) and have permissive licenses. We find the Self-Instruct approach to be less effective at these sizes and propose new ICL methods that draw on two main ideas: (a) categorization and simplification of the ICL templates to make prompt learning easier for the LM, and (b) ensembling over multiple LM outputs to help select high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct seed tasks and employs separate pipelines for instructions that require an input and instructions that do not. Empirical investigations with different LMs show that: (1) Our proposed method yields higher-quality instruction tuning data than Self-Instruct, (2) It improves performances of both vanilla and instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned LMs generate more useful examples than their larger un-tuned counterparts.

## introduction instruction-tuned language models have demonstrated strong zero-shot generalization capabilities to new tasks @xcite @xcite @xcite , creating interest in large-scale automatic synthesis of instructiontuning data @xcite @xcite @xcite . in this context, self-instruct @xcite showed that a small number of expertannotated seed examples, coupled with in-context learning (icl) with a base model, can be used to generate an instruction-tuning dataset to efficiently instruct that same base model. while this method yielded strong results and multiple followup works, most techniques resort to very large lms (around 175b parameters) @xcite , available only through closed-access apis, or have restricted model access. in this paper, we present ensemble-instruct, a novel algorithm enabling high-quality instructiontuning data generation with smaller lms (40b parameters or less), that are also fully accessible and have permissive usage licenses. we show that, when using smaller models as generators, self-instruct struggles to produce text of adequate quality, adversely affecting the utility of the generated data and downstream model performance. staying within the icl framework and using the self-instruct seed tasks, ensemble-instruct explores two main ideas to solve this problem: (1) categorizing and simplifying the icl prompts to ease the few-shot learning process, and (2) ensembling over multiple lm outputs to improve both accuracy and diversity of the generated data. a standard instruction-tuning sample exemplifies a task comprising: (a) an instruction that describes the action to be performed, (b) an optional input on which the action is performed, and (c) the output of the action. similar to self-instruct, we generate samples in two stages: instruction generation and instance generation, where an instance comprises an input (optional) and an output. unlike self-instruct, ensemble-instruct seeks to simplify the problem for the generating lm by first categorizing the examples into two types-those with an input and those without-and then employing separate pipelines for the two that leverage their own unique and simplified prompts ( §2.1). further, it ensembles over the outputs of different lms in two complementary ways: (1) including examples generated by a heterogeneous collection of lms in the final set to increase diversity, and (2) majority voting followed by filtering low-consensus examples to improve accuracy ( §2.4). to understand the effects of our proposed methods, we run an extensive evaluation of different models for instruction generation. this includes vanilla language models (t5) ul2-20b @xcite , falcon-40b @xcite , the instruction-tuned models flan-t5-11b @xcite and flan-ul2-20b @xcite and the chat-tuned foot_0 version of gpt-neox-20b @xcite . as base models to fine-tune with our generated data, we use the vanilla lm pythia-1.4b @xcite for ablation analysis, mpt-7b foot_1 , a decoder only lm similar to llama @xcite as well as gpt-jt-6b foot_2 , an instructed version of gpt-j (wang and komatsuzaki, 2021) trained on chain of thought and natural instruction datasets among others. all chosen models are open-source and have permissive licenses (apache-2). we evaluate the models fine-tuned on the data generated by ensemble-instruct on the the super-natural instructions (superni) test set @xcite and 252 user-oriented tasks from @xcite . our contributions can be summarized as follows: algorithm 1 output ensembling input: lm outputs o 1 , o 2 , o 3 ; threshold t output: best output o best 1: o best ← none 2: rs ← ϕ 3: for (i, j) in {(1, 2), (1, 3), (2, 3)} do 4: rs ← rs ∪ rougel(o i , o j ) 5: end for 6: if min(rs) > t then 7: i, j ← argmax(rs) 8: o best ← o i 9: end if 10: return o best a high-level overview of ensemble-instruct is given in figure 1 . the algorithm has three main components: (i) categorization of tasks and their associated prompts, (ii) generation of instructions followed by instances, where an instance comprises an input (optional) and an output, and (iii) ensemble of outputs from multiple lms. ## analysis of instruction tuning dataset we generate multiple instruction-tuning datasets using a heterogeneous set of lms. ## experimental results we carry out automatic evaluations using rouge-l on the evaluation sets in ## --------------------------------------------------- we set aside 106 tasks (10, 589 samples) from the superni 756 training tasks as the validation data set. for superni instruction tuning, we exclude the validation set from training to simulate evaluation on unseen tasks. we fine-tune 2 base lms on the instruction tuning data generated by the current technique: (1) a vanilla lm, mpt-7b, and (2) an instruction tuned lm, gpt-jt-6b. 9 to fine-tune these models, we adopt qlora @xcite , which enables us to train both lms with a single a100 gpu (40gb memory) within 24 hours. we also carried out full fine-tuning of mpt-7b for 2 data sets, eo-{ul2,neox}-ilm and superni with 2 a100 gpus (80gb memory). the results are shown in tables 7 and 8 for the superni test set, and in in in in gpt-jt-6b 10.4 43.1 mpt-7b 16.6 46.4 open-llama-13b 11.9 46.7 mpt-30b 12.2 49.5 falcon-40b 12.7 49.9 ## related work this work is directly related to self-instruct @xcite , borrowing from it the initial seed tasks and the idea of using icl for tuning a base model into a instruction following model. it could also be seen as related to follow-up works such as: alpaca @xcite )-a practical application of self-instruct-evol-instruct @xcite , which iteratively evolves instructions into increasing difficulty levels and dromedary @xcite , which combines self-instruct with principle-based correction, similar to constitutional ai @xcite . one fundamental limitation of these approaches is that they resort to very large language models (around 175b parameters or 65b parameters at the minimum) that are also proprietary and non-public. here we explore techniques for generating instruction tuning data using lms that are much smaller (around 10b-40b parameters) and have permissive licenses. we crucially draw on a heterogeneous mixture of smaller lms to generate diverse outputs and then ensemble over multiple outputs to select high-quality synthetic examples, while also simplifying the instruction creation process. the use of a reference metric, such as rouge-l, to ensemble the outputs of multiple language distributions is a common technique in minimum bayesian risk decoding, with applications to speech-to-text @xcite , machine translation @xcite , language modeling @xcite and parsing @xcite , among others. here we use a similar technique in the context of instruction generation. to the best of our knowledge, this is the first application of such an approach to instruction-tuning data generation. @xcite proposes llm-blender, an ensembling framework to improve the generaion qualities by leveraging the diverse strengths of multiple language models. while we utilize the output ensemble in the context of synthetic data generation with rouge-l as the reference metric, llm-blender focuses on improving model output qualities using pairranker and genfuser, both approaches capitalize on the efficacy of ensembling as a way of improving output qualities. also related to this work are approaches directly distilling from chatgpt or gpt-4 (openai, 2023) without specific instruction strategies, such as vicuna foot_7 , which distills chatgpt, baize @xcite , distilling conversations and orca @xcite , which uses a large amount of chat-gpt and gpt-4 outputs and combines flan tasks, system prompts and machine-generated explanations sampled from these models. the strength of these approaches seems to rely more on the amount and quality of teacher samples available than on the inductive biases of the self-instructing technique and still rely on proprietary models with non-permissive licenses. ## conclusion we present a novel technique to generate instruction-tuning data through icl, following the recent self-instruct work @xcite . unlike self-instruct, we propose techniques that explicitly avoid the use of proprietary language models like gtp-3, chatgpt or gpt-4. we show that when using smaller models, self-instruct becomes less performant. to overcome this, we draw on two main ideas: (a) categorization and simplification of icl templates to make prompt learning easier, and (b) ensembling over multiple lm outputs to select high-quality examples. these ideas allow us to outperform training with self-instruct while utilizing the same seed tasks. the resulting synthetic data enables base models like mpt-7b to outperform gpt-3, a far larger model with 175b parameters. the results of this work also encourage the departure from closed-access models for advancing instruction generation algorithms.