DATA - CUBE : Data Curriculum for Instruction-based Sentence Representation Learning (2024)

Abstract: Recently, multi-task instruction tuning has been utilized to improve sentence representation learning (SRL). It enables SRL models to generate task-specific representations with the guidance of task instruction, thus exhibiting strong generalization ability on unseen tasks. However, these methods mostly neglect the potential interference problems across different tasks and instances, which may affect the training of the model.To address this issue, we propose a data curriculum method, namely **Data-CUBE**, that arranges the order of all the multi-task data for training, to minimize the interference risks from two aspects.At the task level, we aim to find the optimal task order to minimize the total cross-task interference risk and formulate this problem as the traveling salesman problem, which is further solved by a specially designed simulated annealing algorithm. At the instance level, we propose a measurement method to quantify the difficulty of all instances per task, and then arrange instances in an easy-to-difficult order for training.Experimental results show that our approach can boost the performance of state-of-the-art methods. Our code and data will be publicly released.

## introduction sentence representation learning (srl) @xcite is a fundamental task in the nlp field, which focuses on encoding the semantic information of sentences into low-dimensional vectors. typically, existing work @xcite collects a set of sentence pairs (or augmented in an unsupervised way), and then learns the model parameters by maximizing and minimizing the similarity scores of relevant and irrelevant sentences, respectively. previous srl methods based on advanced language models and learning objectives @xcite task 1 task 4 task 2 task 3 instance difficulty high-risk task order task 2 data the underfitting degrees of all training tasks. according to the ratio of instances whose positives and negatives are not clearly distinguished (@xmath0.05), we categorize all tasks into three degrees: severe (>80%), moderate (>50% @xmath1, and mild (<50%). @xcite , are capable of producing high-quality representations that perform well on various downstream tasks. despite the success, recent studies @xcite have revealed that it is challenging to directly transfer the learned sentence representations into new tasks, even causing significant performance degradation. to alleviate this problem, instruction tuning @xcite has been applied to sentence representation learning, which collects a diverse set of sentencepair datasets with task-specific natural language instructions @xcite . before training, each of the collected datasets is generally divided into multiple mini-batches at random @xcite , and the srl model will be trained over the mini-batches of all datasets in a random order. after multi-task training on the dataset collection, the model would become capable of generating task-specific sentence representations with the guidance of the task instruction, exhibiting improved generalization ability on unseen tasks. however, as the collected datasets vary in data distributions, a random order for instruction data scheduling would lead to potential cross-task interference risk for model optimization. as depicted in figure 1 (left), when the neighboring mini-batches are from very different tasks, the successive learning of them would lead to conflict in the optimization objective, affecting the final performance of both tasks. in addition, the instances for a given task might be with varied difficulty levels. randomly assigning them into mini-batches may result in potential cross-instance interference, which is also likely to cause the performance degradation. as shown in figure 1 (right), the competitive model instructor @xcite struggles with distinguishing more than 80% positive and negative examples in almost half of the training datasets, indicating the severe underfitting problem. to address these issues, in this paper, we propose data-cube, a data curriculum method for instruction-based sentence representation learning. the core idea of our approach is to design a proper data curriculum that arranges the orders of all tasks and instances for minimizing the potential interference risks. concretely, for the crosstask interference, we focus on finding the optimal task order where all the neighboring two tasks are as similar as possible, to minimize the total interference risks derived from task divergence. we formulate this task as the the travelling salesman problem (tsp) @xcite : all the tasks are regarded as the nodes in a fully connected graph with task similarity as the edge weights, and the optimal order search problem is essentially to find the longest route that visits all nodes. to solve the tsp, we employ the widely-studied simulated annealing algorithm to efficiently find its suboptimal solution. for the cross-instance interference, we measure the discriminability of positive and negative, as the estimated difficulty for sorting all the instances. then, we divide them into easy-to-difficult minibatches for training, to minimize the interference risks caused by varied instance difficulty. to integrate the two strategies, we first find the optimal task order with the simulated annealing algorithm, then sort the instances within each task according to their difficulty to obtain the instance order for mini-batch arrangement, and finally divide the sorted instances from all the tasks into a sequence of mini-batches for training srl models. our contributions are summarized as follows: instruction tuning. instruction tuning @xcite involves supervised fine-tuning pre-trained language models by integrating well-formatted natural language instructions into the input. this process is closely connected to multi-task learning and is believed to enhance the generalization capability of language models across a range of tasks @xcite . previous studies have demonstrated that increasing the number and diversity of tasks associated with instructions can improve performance. considering the effectiveness of instruction tuning, it has been applied to various nlp tasks, such as sentence representation learning @xcite . however, with the increasing diversity of tasks, there is a potential for interference across different tasks, which may lead to performance degradation @xcite . hence, we propose to leverage data curriculum to alleviate the interference in the multi-task instruction tuning. ## approach in this section, we present the proposed data-cube, a data curriculum method for instruction-based sentence representation learning. following previous work, we develop the approach based on a multi-task contrastive learning framework, and while introduce novel data curriculum methods for srl, considering both task-level and instance-level arrangement, which can significantly reduce the learning interference issue. ## experiments in this section, we train with data-cube based on instructor and conduct evaluations on four task categories within mteb @xcite , encompassing a total of 28 downstream tasks. furthermore, we continue to investigate the effectiveness and robustness of data-cube. for detailed settings, see appendix b and c. ## conclusion in this work, we proposed data-cube, a data curriculum method for multi-task instruction based sentence representation learning. the key idea of our approach is to reduce the cross-task and crossinstance interference risks by using a more suitable data curriculum of instances for training. to achieve this, we employed a simulated annealing algorithm to find the optimal task order to minimize the cross-task interference, and assigned all instances per task into easy-to-difficult mini-batches to reduce the cross-instance interference. experimental results on mteb sentence representation evaluation tasks have shown that our approach can boost the performance of state-of-the-art baselines.