RECOST : External Knowledge Guided Data-efficient Instruction Tuning (2024)

Abstract: In the current landscape of large language models (LLMs), the process of instruction tuning serves as an essential step. Considering the high computing power overhead, data-efficient instruction tuning was proposed to reduce the training data size in this process, aiming at selecting high-quality instructional data. Nevertheless, we argue that most current data-efficient instruction-tuning methods are highly dependent on the quality of the original instruction-tuning dataset. When it comes to datasets synthesized by LLMs, a common scenario in this field, dirty samples will even be selected with a higher probability than other samples. To address these challenges, we utilized external knowledge (relevant examples or paragraphs) to evaluate those samples synthesized by LLMs with an in-context-based relative predictive entropy. Based on the new metric, we proposed a framework, dubbed as RECOST , which integrates external-knowledge-base re-ranking and diversity-consistent sampling into a single pipeline. Through extensive experiments on several synthetic datasets (Alpaca and Alpaca-gpt4), we demonstrate the effectiveness of our method and achieve even better results with only 1% of the full dataset.

## introduction large language models (llms) @xcite have demonstrated their remarkable capabilities in numerous fields of natural language processing (nlp) with the advancing of training datasets and the scale of model parameters. behind this phenomenon, instruction tuning serves as an essential step to help pre-trained llms align to human cognition @xcite @xcite . instruction tuning refers to fine-tuning the llms on instruction-response pairs to endow llms with instruction-following capability and activate the knowledge gained in the pre-training period. in the past two years, three types of instructionfollowing datasets have emerged @xcite : those based on traditional nlp tasks (eg. flan v2 @xcite ), those based on high-quality manual annotation (eg. lima @xcite ), and those synthesized by llms (eg. alpaca @xcite ). among these, lima asserts that the quality of instructionfollowing datasets is far more important than their quantity. thus, data-efficient instruction tuning is proposed to reduce the data size in instruction tuning without compromising the models' performance @xcite . contemporary works in data-efficient instruction tuning predominantly concentrate on selecting high-quality data from instruction datasets synthesized by llms @xcite @xcite . most approaches involve evaluation based on its proposed metrics, primarily centered around metrics related to predictive entropy @xcite . we simply conclude these methods to synthetic-knowledge-guided ones as they select data points by using synthetic data points as prior knowledge. however, duan et al. figured out that predictive entropy is not reliable enough. as shown in figure 1 , selection based on predictive entropy still results in the sampling of a non-negligible proportion of noisy data. what's even more surprising is that the higher the quality ranking, the greater the probability that samples contain noise, which is completely contrary to the original design intention. we argue that these methods heavily rely on the quality of the curated datasets which serve as the pre-experience @xcite or test set @xcite in their processes and are susceptible to the influence of outlier samples within the original datasets. therefore, in data selection under synthetic datasets scenarios, we argue that synthetic-knowledge-free methods are still underexplored. this issue currently imposes significant limitations on the development of the field of dataefficient instruction-tuning. to address the challenge posed by the limitations of the predictive entropy in vanilla llms as outlined above, we utilize external information to evaluate samples synthesized by llms. despite the suboptimal performance of this dataset in generative tasks @xcite , its authenticity is significantly assured. but in the data-efficient instruction-tuning scenario of llm, this cost is unacceptable. recognizing the importance of maintaining efficiency, we instead intuitively leverage pre-trained llms' intrinsic in-context learning (icl) capabilities, treating these truthful samples as demonstrations. building on this foundation, we introduce a concept: in-context-knowledge-based relative predictive entropy, which serves as another dimension of uncertainty for vanilla llms. in this paper, we propose recost (retrieval, re-rank, coreset sampling, and supervised fine-tuning), a framework that encompasses an incontext-knowledge-based re-ranking module and a diversity-consistent sampling module to avoid an overly homogeneous data distribution after reranking. with extensive experiments on synthetic datasets including alpaca and alpaca-gpt4, re-cost demonstrates its superiority over previous methods and surpasses remarkably the full-trained model with merely 1% and 10% training data on three benchmarks including the alpagasus test sets @xcite , the open-llm benchmark @xcite and alpacaeval @xcite all in all, our work explores how to instruct-tune llms under data-efficient scenarios with synthetic datasets. our contributions can be summarized as follows: ## methodology in this section, we will dive into the main methodology of recost. figure 2 briefly illustrates the framework of our method. ## experiments results and analysis in this section, we present the main results of our method on the three benchmarks mentioned in section 4.2. moreover, we conduct extensive ablation studies based on several possible factors. ## conclusion in this paper, we demonstrate recost, an effective method to select high-quality instruction data from synthetic instruction datasets with the help of external knowledge. our method outperforms the fully-trained models with only 1% training data, which surpasses the previous methods under the same settings. overall, our approach underscores the significance and efficacy of integrating our proposed relative uncertainty into dataefficient instruction tuning for synthetic datasets, providing a viable avenue for this field.