A da-Instruct: Adapting Instruction Generators for Complex Reasoning (2024)

Abstract: Instructions augmentation is a crucial step for unleashing the full potential of large language models (LLMs) in downstream tasks. Existing Self-Instruct methods primarily simulate new instructions from a few initial instructions with in-context learning. However, our study identifies a critical flaw in this approach: even with GPT4o, it cannot generate complex instructions of length \ge 100 , which is necessary in complex tasks such as code completion.To address this issue, our key insight is that fine-tuning open source LLMs with only ten examples can produce complex instructions that maintain distributional consistency for complex reasoning tasks. We introduce Ada-Instruct, an adaptive instruction generator developed through fine-tuning. We empirically validated Ada-Instruct’s efficacy across different applications. The results highlight Ada-Instruct’s capacity to generate long, intricate, and distributionally consistent instructions.

## introduction supervised fine-tuning (sft) is crucial for harnessing the potential of pre-trained large language models (llms) in downstream tasks. addressing sft's demand for extensive training data, recent research has employed advanced llms, such as chatgpt, to generate instructions. a prevalent approach is called "self-instruct" @xcite , which involves having chatgpt sequentially generate both instructions and answers @xcite @xcite @xcite @xcite . it efficiently generates substantial novel training samples from a minimal number of initial samples. however, our observations reveal a fundamental and critical limitation of self-instruct -it notably struggles to generate complex instructions. despite being demonstrated with long and complex examples, self-instruct predominantly produces disappointingly brief and overly simplistic instructions. this is evident in figure 1 (a) and figure 1(d ), where we present the length distribution of instructions by self-instruct on humaneval (programming) and gsm8k (mathematics). the figures expose a glaring gap: self-instruct fails to produce instructions that exceed 100 and 60 tokens for hu-maneval and gsm8k, respectively. this limitation significantly undermines the use of self-instruct in more complex tasks. is prompt engineering a solution? despite its widespread use in enhancing in-context learning, prompt engineering is not the panacea it is often made out to be @xcite @xcite . to encourage the generation of longer and more complex instructions, we explored infusing prompts with extra requirements, such as "generate algorithms of intermediate level" (for humaneval) and "the instructions should not be too easy" (for gsm8k). however, as shown in figure 1(b) 1(e), this approach did not effectively solve the problem of producing short instructions. a more advanced variant of prompt engineering, evol-instruct, employs multiturn strategies to incrementally enhance the complexity and variety of instructions. however, we will show in § 4.4.1 that evol-instruct is unable to generate instructions that semantically align with the target instruction distribution. has the problem been solved by the more advanced gpt4o? we performed additional evaluations using the advanced gpt4o model, which is equipped with superior reasoning and long-text processing capabilities. figures 1(c )1(f) illustrate that while gpt4o outperforms gpt-3.5-turbo-instruct in terms of average output length on the humaneval benchmark, it still falls short of generating instructions longer than 100 tokens. similarly, on the gsm8k benchmark, gpt4o shows no marked im- in this paper, we unveil a novel insight into the instruction generation capabilities. surprisingly, we find that even when relying solely on 10 samples, a straightforward fine-tuned model is capable of generating instructions that align with the target task distribution. in figure 1 (g), ft models generate instructions of @xmath0, and in figure 1 (h), of @xmath1, both matching the actual distribution. in addition, the generated instructions span the target distribution ( § 4.4.1), and exhibit high diversity ( § 4.4.2). based on these findings, we introduce ada-instruct, a few-shot instruction generation procedure for downstream tasks. we fine-tune opensource llms using few-shot task samples for instruction generation, instead of icl as in self-instruct. in summary, our contributions include (1) we uncover a new insight into the sample generation capabilities of self-instruct, showing that it cannot generate complex instructions. (2) we introduce ada-instruct, a few-shot instruction generation methodology with fine-tuning. (3) we verify the effectiveness of ada-instruct through empirical validations, showcasing its superiority in generating complex instructions that are not only longer, but also aligned with the target distributions. 2 related work sample generation via llms recent works have explored the use of llms for sample generation, often within the self-instruction framework @xcite . this typically involves starting from an initial pool of instructions and having the llms iteratively generate new instructions along with the corresponding answers. most prior work in the realm of instruction generation has relied on icl @xcite @xcite @xcite . various studies have focused mainly on improving the self-instruct approach in different problem scenarios. however, a limitation of this paradigm, as we have observed, is that icl lacks the capacity to generate complex samples based solely on in-context examples. although more intricate samples could potentially be produced using evolutionary strategies, such as evol-instruct @xcite , these manually designed tactics risk generating samples that do not align with the target task distribution. fewgen @xcite is the only method we have identified that substitutes fine-tuning for in-context learning (icl) in sample generation. however, fewgen requires sophisticated metalearning and is limited to classification tasks. in contrast, ada-instruct is substantially simpler and more general. icl vs. ft previous exploratory studies have aimed to compare the performance of icl and ft methodologies. some research suggests that icl exhibits a more robust out-of-distribution generalization compared to ft @xcite @xcite . however, some recent studies @xcite argue that these earlier comparisons may be biased. the unfairness arises from using different model architectures for comparison (e.g., gpt-3-based icl versus roberta @xcite -based ft) or by basing results on small-scale models. in more equitable experimental setups, the researchers found that ft outperforms icl @xcite , thereby supporting our strategy of using ft models for instruction generation. ## method ada-instruct is divided into three steps: 1) learning an instruction generator and generating massive instructions ( § 3.1), 2) generating labels with chat-gpt ( § 3.2), and 3) training llms for downstream tasks ( § 3.3). in the following, we dive into the details of each step. the overall workflow is shown in figure 2 . ## conclusion we unveil novel insights into the capabilities of instruction generation, demonstrating that the conventional icl-based self-instruct fails to generate long and complex instructions. in contrast, we reveal the proficiency of fine-tuning in generating task-aligned instructions, even with a limited number of initial samples. we introduced ada-instruct, a novel few-shot instruction generation methodology that leverages the fine-tuning of open-source llms, diverging significantly from the prevalent self-instruct strategies based on in-context learning with closed-source llms. ada-instruct ensures the generation of coherent, high-quality, and di-verse instructions that align well with the target task distribution, presenting a groundbreaking solution to the challenges of data sparsity and diversity in instruction generation.