Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLM s: A Case Study in T hai (2024)

Abstract: We present a synthetic data approach for instruction-tuning large language models (LLMs) for low-resource languages in a data-efficient manner, specifically focusing on Thai. We identify three key properties that contribute to the effectiveness of instruction-tuning datasets: fluency, diversity, and cultural context. We propose a seed-data-free framework for generating synthetic instruction-tuning data that incorporates these essential properties. Our framework employs an LLM to generate diverse topics, retrieve relevant contexts from Wikipedia, and create instructions for various tasks, such as question answering, summarization, and conversation. The experimental results show that our best-performing synthetic dataset, which incorporates all three key properties, achieves competitive performance using only 5,000 instructions when compared to state-of-the-art Thai LLMs trained on hundreds of thousands of instructions. Our code and dataset are publicly available at https://github.com/parinzee/seed-free-synthetic-instruct.

## introduction large language models (llms) have achieved a near human-level of performance across multitudes of tasks and domains @xcite @xcite . however, many evaluation results have shown that this level of performance is often limited to high-resource languages only, with inconsistent levels of performance for lower-resource languages, i.e., thai @xcite @xcite @xcite . the development of llms for lowresource languages is crucial for enabling impactful applications for millions of people worldwide. some applications of these llms include medical chatbots @xcite , intelligent tutoring systems @xcite , and content moderation tools that could help combat misformation and hate speech @xcite . these potential applications have motivated researchers to explore methods for improving llm performance in low-resource languages. recently, researchers developed fine-tuning techniques to improve the performance of llms in thai as well. sambalingo @xcite investigated how performing continual pretraining and instruction-tuning on machine-translated english datasets results in good performance in multiple low-resource target languages including thai. wangchanx @xcite explored using pre-existing thai datasets to perform instruction-tuning and adapt the sea-lion model @xcite to the thai language. typhoon-instruct @xcite adapted llama-3 (ai@meta, 2024) to the thai language through continual pretraining on a filtered web corpus, and instruction-tuning on a combination of machine-translated datasets and thai synthetic datasets generated with @xcite . these methods typically use over 50k and sometimes over 100k examples for their instructiontuning process, making it very costly. additionally, some of these approaches involve continual pretraining, which further increases the cost and complexity of the model development process. it remains unclear whether such large datasets are truly necessary for achieving high performance in low-resource languages, as the aforementioned works in thai llms do not address this. however, other works have also shown that llm alignment in english does not require extensively large datasets @xcite . thus, by carefully designing a high-quality synthetic dataset tailored to the target language, we hypothesize that it may be possible to achieve similar performance improvements in thai while significantly reducing the data requirements and associated costs. to formulate a high-quality synthetic dataset, we identify three key properties that the datasets used to finetune the current thai llms have: (1) fluency: the data is grammatically correct and natural-sounding, enabling the model to learn the proper structure and flow of the language. (2) diversity: the data consists of a wide range of topics and domains, allowing it to generalize better to various downstream tasks. (3) cultural context: the data contains instructions and information relating to the culture and beliefs appropriate for the average person from the country of the target language. these properties are commonly present within the dataset used to train these models. fluency is inherently present in the way that humans write and thus is within the human-annotated han instruct dataset @xcite used to train wangchanx. diversity comes from the fact that existing datasets commonly cover multiple domains. for example, sambalingo uses a translated version of ultrachat, which covers a wide range of topics. cultural context is also present in these datasets. for example, iapp wiki qa (viriyayudhakorn and polpanumas, 2021)-a subset of openthaigpt's training dataset-includes questions on thai wikipedia data. we hypothesize that combining all three properties in a dataset will yield a reasonably performant thai llm, even if the dataset is synthetic. to verify our hypothesis, in this paper, we develop a framework to generate synthetic instructiontuning datasets with controllable parameters for each of these properties. we use our framework to create five datasets with varying combinations of the properties as detailed in section 4.1. we then perform instruction-tuning with the base model of llama-3 8b (ai@meta, 2024) on each dataset and evaluate their performance on two benchmarks: culture-specific and non-culture-specific datasets. our findings suggest that incorporating all three properties in the training data improves the performance of llms in low-resource languages, and using only just 5k rows of our dataset for instructiontuning allows for similar performance against other methods that used 10-100x larger datasets. we summarize the contribution of our work as follows: ## conclusion and future work in conclusion, this study demonstrates the effectiveness of a data-centric approach for improving the performance of large language models in thai, a low-resource language. we identified three key properties that lead to well-performing thai llms: fluency, diversity, and cultural context. we proposed a seed-data-free framework for generating high-quality instruction-tuning data that incorporates these properties. experiments conducted across multiple models, synthetic datasets, benchmarks, and tasks provide empirical evidence that it is possible to achieve competitive results compared to state-of-the-art thai llms trained on 10-100x larger datasets. while our model tends to generate more concise responses compared to the topperforming typhoon-instruct model, impacting its performance on open-ended generative tasks, it still achieves impressive results overall, beating other models such as openthaigpt-v1.0.0 and achieving comparable results to wangchanx llama3-8b sft demo. for future work, there are several promising directions to explore. one important avenue is to extend our pipeline to generate multi-turn dialogue datasets, which can help alleviate the length issues observed in the current study and enhance the model's ability to handle more realistic, conversation-based scenarios. additionally, conducting experiments with a stronger base model, such as upgrading from claude-3 haiku to a more advanced model, could potentially yield even better performance without requiring significant modifications to the data generation process. to assess the generalizability of our approach, it would be valuable to expand our experiments to other lowresource languages, adapting the framework to handle different linguistic properties and cultural contexts.