Mosaic- IT : Cost-Free Compositional Data Synthesis for Instruction Tuning (2025)

Abstract: Finetuning large language models with a variety of instruction-response pairs has enhanced their capability to understand and follow instructions. Current instruction tuning primarily relies on teacher models or human intervention to generate and refine the instructions and responses for training, which are costly, non-sustainable, and may lack diversity. In this paper, we introduce Mosaic Instruction Tuning (Mosaic-IT), a human/model-free compositional data synthesis method that can efficiently create rich and diverse augmentations from existing instruction tuning data to enhance the LLMs. Mosaic-IT randomly concatenates multiple instruction data into one and trains the model to produce the corresponding responses with predefined higher-level meta-instructions to strengthen its multi-step instruction-following and format-following skills. Our extensive evaluations demonstrate a superior performance and training efficiency of Mosaic-IT, which achieves consistent performance improvements over various benchmarks and an 80% reduction in training costs compared with original instruction tuning.

## introduction the emergence of large language models (llms) @xcite @xcite along with their remarkable performance in downstream tasks @xcite , has revolutionized the domains of artificial intelligence and natural language processing. a key component of the recipe to unlock the exceptional ability of llms in understanding and following instructions is the technique of instruction tuning (it) @xcite @xcite , which involves the fine-tuning of llms on datasets comprising corresponding instruction-response pairs. figure 1 : the brief illustration of our mosaic-it with different strategies. given the original dataset, our method randomly samples and concatenates them together into more complex samples, simulating the multiinstruction-following scenarios in a cost-free manner. to ensure the quality of instruction tuning data, earlier efforts @xcite @xcite carefully curate extensive, diverse, and high-quality datasets manually. although these datasets encompass a wide range of instructions to improve instruction tuning, they require the responses to be meticulously curated by human experts @xcite @xcite @xcite . alternatively, some approaches @xcite @xcite leverage more capable teacher llms to reduce the labor-intensive process of data generation. for example, the alpaca @xcite utilizes self-instruct @xcite to automatically generate diverse instruction tuning datasets. building on this trend and the widely acknowledged notion that more complicated instructions are more beneficial for llms' instruction-following ability @xcite , numerous strategies @xcite @xcite @xcite @xcite have been proposed to further diversify and complexity the instruction-response pairs, utilizing teacher models like @xcite . despite the enhanced performance in instructionfollowing ability offered by these existing methods, they face two major issues: (1) they heavily rely on teacher models or human annotators to rewrite instruction-response pairs, which highlights the resource-intensive nature and their constraints on scalability; (2) they only increase the complexity within the scope of a single instruction, which limits the potential improvement in llms' instruction-following capabilities. motivated by the dense and aligned captions @xcite proposed for vision language (vl) models and the mosaic data augmentation proposed in @xcite , we hypothesize that denser instructions benefit the llm alignment, i.e. the process of instruction tuning should not be constrained by one single instruction but be extended to follow several instructions at a time, which represents a higher level of instructionfollowing ability that is beneficial to the training process. a similar concept during the inference phase is proposed by batch prompting @xcite , where multiple samples are grouped in one batch, allowing llms to generate multiple responses at one inference, while its performance is sub-optimal. as orthogonal to the existing instruction tuning methods, we introduce mosaic instruction tuning (mosaic-it), an innovative and model/human-free compositional approach that augments existing instruction tuning datasets, which concurrently improves the llm performances and lowers the training expenses. as shown in figure 1 , in our method, multiple instructions and corresponding responses from the original dataset are concatenated into a single sample for fine-tuning, simulating the multi-instruction-following scenarios at no cost. without applying any additional strategies, we term this simple process as the primary mosaic strategy. we posit that this mosaic strategy process significantly improves the complexity and density of the original instructions, learning from which directly benefits llms in their instructionfollowing ability. additionally, this method offers the advantage of directly reducing the total count of instruction-response pairs, thereby cutting down on training iterations, and accelerating the training process significantly by approximately 80% reduction. though effective, the primary mosaic strategy constrains llms in responding to the instructions in the original order and format, potentially limiting their further potential. thus, we further introduce three advanced mosaic strategies aimed at enhancing the diversity and complexity of the mosaicked instruction-response pairs: format, permute, and maskout, in which an additional metainstruction is provided as a higher-level guideline for llms to follow the given instructions. illustrative examples are presented in figure 2 . specifically, in the format strategy, some arbitrary parsing formats will be defined in the meta-instruction, thus forcing llms to follow these formats, which notably enhances the llms' capacity to follow formats. in the permutation strategy, an arbitrary permuted order is defined, thus forcing llms to respond in a desired order. in the maskout strategy, some arbitrary instructions are sampled, which meta-instruction forces llms to ignore. moreover, the use of these advanced strategies not only boosts the performance in several evaluation metrics but also keeps our method free of additional costs. in summary, our primary contributions can be illustrated as follows: ## conclusion we introduce mosaic instruction tuning (mosaic-it), a novel, human/model-free method to enhance instruction tuning for llms. by concatenating multiple instruction-response samples and using higher-level meta-instructions, mosaic-it improves multi-step and format-following capabilities. our evaluations show superior performance and an 80% reduction in training costs compared to the original methods. mosaic-it's simplicity and efficiency make it a scalable solution for improving llms without extensive human intervention or resource-intensive teacher models. our results highlight the potential of innovative data synthesis techniques in advancing llm capabilities. identify some of the main components of a computer network such as hosts, routers, switches, hubs, links, and interfaces. explain how they communicate and coordinate with each other using protocols and standards such as tcp/ip, http, ftp, dns, dhcp, and arp. describe how data is transmitted and received over a network using concepts such as packets, frames, headers, addresses, ports, sockets, and checksums. a computer network is composed of several components that work together to facilitate communication and data transfer between devices. these components include: 1. hosts: a host is a device that is connected to the network, such as a computer, smartphone, or printer. each host has a unique ip address, which is used to identify it on the network. 2. routers: a router is a device that connects multiple networks together and directs traffic between them. it uses protocols such as ip to route data packets to their destination. 3. switches: a switch is a device that connects multiple devices together on a single network. it uses mac addresses to direct data packets to their destination. 4. hubs: a hub is a device that connects multiple devices together on a single network. it broadcasts data packets to all connected devices, which allows them to communicate with each other. 5. links: a link is the physical connection between two devices on a network. it can be a cable, a wireless connection, or a combination of both. 6. interfaces: an interface is the part of a device that connects it to the network. it can be a network card, a usb port, or a wireless card. all of these components communicate and coordinate with each other using protocols and standards such as tcp/ip, http, ftp, dns, dhcp, and arp. these protocols define how data is transmitted and received over a network, how devices are addressed, and how traffic is routed. data is transmitted over a network in packets or frames. each packet or frame contains a header, which includes information such as the destination ip address, the source ip address, and the type of data being transmitted. the data is then transmitted over the network, and the receiving device uses the information in the header to identify the data and direct it to the appropriate application or program. in summary, a computer network is composed of several components that work together to facilitate communication and data transfer. these components communicate and coordinate with each other using protocols and standards, and data is transmitted over the network in packets or frames, using headers to identify the destination, source, and type of data. a computer network is made up of several components that work together to facilitate communication and data transfer between devices. some of the main components include: 1. hosts: these are the devices that are connected to the network and use its resources, such as computers, printers, and servers. 2. routers: routers are devices that direct traffic between networks, and are used to connect multiple networks together. 3. switches: switches are used to direct traffic within a network, and are used to connect multiple devices together on a single network. 4. hubs: hubs are devices that connect multiple devices together on a single network, and are used to distribute data to multiple devices at once. 5. links: links are the physical connections between devices on a network, such as ethernet cables or wireless connections. 6. interfaces: interfaces are the points of connection between the network and the devices on it, such as network cards or wireless adapters. for these components to communicate and coordinate with each other, they use a variety of protocols and standards. protocols are sets of rules that dictate how data is transmitted and received, while standards are agreed-upon methods or formats that allow for interoperability between different systems. some common protocols and standards used in computer networking include: -tcp/ip: this is a suite of protocols that is used to communicate between devices on a network. it includes protocols for addressing, routing, and transporting data. when data is transmitted over a network, it is broken down into small packets, each of which contains information about where it needs to go and what it contains. these packets are sent over the network, hopping from one device to the next until they reach their destination. at each hop, the device examines the packet's header to see where it needs to go next, and then forwards it along the appropriate link. once the packet reaches its destination, the receiving device examines the header to see if it is the intended recipient. if it is, the device will extract the data from the packet and use it, otherwise, the packet will be discarded. to ensure that the data in the packets is not corrupted during transmission, checksums are used. a checksum is a mathematical calculation performed on the data in the packet, which produces a value that can be used to verify that the data has not been altered during transmission. if the checksum of the data in a packet is incorrect when it arrives at its destination, it is discarded, and the sender is notified of the error. in summary, data is transmitted over a network by being broken down into small packets, each of which contains information about where it needs to go and what it contains. these packets are sent hopping from one device to the next, using a variety of protocols and standards to coordinate their transmission and reception. checksums are used to ensure that the data in the packets has not been corrupted during transmission, and any errors are detected and corrected.