G - DIG : Towards Gradient-based DI verse and hi G h-quality Instruction Data Selection for Machine Translation (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable abilities in general scenarios. Instruction finetuning empowers them to align with humans in various tasks. Nevertheless, the Diversity and Quality of the instruction data remain two main challenges for instruction finetuning. With regard to this, in this paper, we propose a novel gradient-based method to automatically select high-quality and diverse instruction finetuning data for machine translation. Our key innovation centers around analyzing how individual training examples influence the model during training. Specifically, we select training examples that exert beneficial influences on the model as high-quality ones by means of Influence Function plus a small high-quality seed dataset. Moreover, to enhance the diversity of the training data we maximize the variety of influences they have on the model by clustering on their gradients and resampling. Extensive experiments on WMT22 and FLORES translation tasks demonstrate the superiority of our methods, and in-depth analysis further validates their effectiveness and generalization.

## introduction large language models (llm) have revolutionized the field of natural language processing with their strong abilities in general language understanding and generation @xcite . to enable this strong ability, instruction finetuning has been proposed to better align language models @xcite @xcite . significant progress has been made on collecting extensive instruction finetuning data for better aligning llms to produce helpful responses @xcite . we argue that diversity and quality of the instruction data present a pair of challenges for instruction finetuning. @xcite have demonstrated that a model trained with a limited, carefully curated dataset composed of high-quality and diverse examples outperforms models trained on larger, more extensive datasets during instruction finetuning. subsequently, various methods have been proposed to automatically select high-quality and diverse training data from the large pool of instruction finetuning datasets @xcite . yet, these methods often rely on another model to decide quality or diversity, neglecting the inherent model behavior and strong ability of the llm itself. to this end, we propose g-dig, a novel gradient-based method to automatically select diverse and high-quality instruction finetuning data for machine translation. we use influence function @xcite , a gradient-based method that quantifies the impact made by individual training samples. concretely, we (1) measure the response quality of each training sample with the influence score of the training sample on test instances and (2) enhance the diversity of the training data by maximizing the variety of influences they have on the model. specifically, we hypothesize that high-quality data should have positive influences on high-quality test samples. hence, we first manually create a small set of high-quality seed data and then automatically select high-quality data that have positive influences on seed data. moreover, we utilize kmeans clustering algorithms to cluster training data with similar influences, using gradients representing their influences on the model. unlike existing methods that introduce an external model to decide quality and diversity, our methods directly use model gradients, which capture the model behavior through learning algorithms and back to the training data. we conduct experiments on zh ⇒ en and de ⇒ en translation tasks. specially, we collect large candidate pools and manually construct two small sets of seed data. we then finetune different llm backbones on various sizes of selected subsets and compare their performances with different selective methods and existing sota llms. under a thorough comparison in a range from 1k to 64k selected samples, our proposed method not only surpasses the baseline selective methods but also achieves competitive performance against sota models. extensive experiments and in-depth analysis emphasize the need for data selection and demonstrate the effectiveness and generalization of our proposed methods. the aforementioned studies all focus on the inference stage optimization, while another line of work focuses on instruction tuning the llms for better translation performance. @xcite proposes to first finetune the model on monolingual data and then on high-quality parallel data. @xcite trains the model to elicit translation ability by multilingual instruction tuning. @xcite proposes to create high-quality instruction-tuning data from larger models by a patching mechanism. @xcite improves the model instruction understanding by adding a global instruction representation and improves model faithfulness by comparing over-translation and misstranslation results with the correct translation. @xcite proposes a novel framework using examples in comparison to teach llms to learn translation. however, all these methods neglect the importance of instruction finetuning data quality and diversity in machine translation. and in this paper, we propose a novel approach to enhance the quality and diversity of translation data. ## methods in this section, we describe our gradient-based method to select high-quality and diverse training data for instruction finetuning, as displayed in figure 1 . our methods consist of two parts: 1) high-quality data selection with the influence function ( § 3.1) and 2) diverse data selection with gradient-based clustering ( § 3.2). for high-quality data selection, importantly, we utilize the influence function to quantify the impact of individual training on the test sample. for diverse data selection, we use gradient distance to assess the overall diversity of the instruction training data. ## conclusion in this paper, we propose g-dig, a novel gradientbased method for selecting high-quality and diverse llm finetuning data for machine translation. specifically, using influence function and a seed dataset we select high-quality training data that have beneficial influence on the model. furthermore, we enhance the training data diversity by clustering on their gradients and resampling. extensive experiments prove that our methods improve the llm in terms of translation ability. also, human evaluation results demonstrate that our methods better align the llm compared with the baseline model. we hope this work facilitates the research on llm finetuning data selection.