SELF - EXPERTISE : Knowledge-based Instruction Dataset Augmentation for a Legal Expert Language Model (2024)

Abstract: The advent of instruction-tuned large language models (LLMs) has significantly advanced the field of automatic instruction dataset augmentation. However, the method of generating instructions and outputs from inherent knowledge of LLM can unintentionally produce hallucinations — instances of generating factually incorrect or misleading information. To overcome this, we propose SELF-EXPERTISE, automatically generating instruction dataset in the legal domain from a seed dataset. SELF-EXPERTISE extracts knowledge from the outputs of the seed dataset, and generates new instructions, inputs, and outputs. In this way, the proposed method reduces hallucination in automatic instruction augmentation. We trained an SELF-EXPERTISE augmented instruction dataset on the LLaMA-2 7B model to construct Korean legal specialized model, called LxPERT. LxPERT has demonstrated performance surpassing GPT-3.5-turbo in both in-domain and out-of-domain datasets. The SELF-EXPERTISE augmentation pipeline is not only applicable to the legal field but is also expected to be extendable to various domains, potentially advancing domain-specialized LLMs.

## introduction recent large language models (llms) like gpts @xcite , palm @xcite show exceptional performance for various nlp tasks in response to instruction prompt. to train llms in following natural language instructions, an instruction dataset is essential for performing instruction tuning @xcite @xcite . however, acquiring a diverse human-written instruction dataset is often time-consuming and costly. to overcome these limitations, many have used llms for auto-generation @xcite 2023). however, such methods remain unsuitable for creating instruction datasets in domains where accuracy is critical, because llms lack expertlevel knowledge, can produce hallucinated data @xcite @xcite . therefore, high-quality, accuracy-focused methodologies are needed for generating instruction data in specialized knowledge areas to extend the usability of llms beyond general tasks. in this study, we introduce the novel self-expertise method for automatic instruction data generation for knowledge-intensive tasks using llms. it prevents hallucinations and ensuring data accuracy in specialized knowledge do-mains by generating data based on precise knowledge. figure 2 describes four-step process of self-expertise. inspired by how a teacher designs exam questions based on textbook context, user instructions, inputs, and outputs are created based on the knowledge, extracted from small set of seed data outputs. we automatically generated a legal domain instruction dataset of 19k from 980 seed dataset utilizing self-expertise. then, we instruction tuned llama-2 7b @xcite using self-expertise augmented dataset. we refer to this resulting model as lxpert, for legal expert. comparisons of lxpert with models instruction-tuned in general domains and those tuned with datasets generated by traditional augmentation methods reveal its superior accuracy and fluency. furthermore, lxpert significantly surpasses gpt-3.5-turbo, the most widely used model lately. figure 1 shows comparison between gpt-3.5-turbo and lxpert. these findings underscore the importance of specialized models for specific domains @xcite @xcite , highlighting the effectiveness of the self-expertise approach in developing models that deliver high performance in professional fields. in summary, our contributions are as follows: ## methodology we propose self-expertise, a novel methodology for automatically generating instruction data based on knowledge, thus enabling precise logical reasoning that reflects the characteristics of specialized knowledge areas. ## legal self-expertise data we applied the self-expertise methodology to the field of law, where accuracy and reasoning are crucial. we give detailed explanation of the self-expertise augmented instruction dataset. ## discussion in our experiments with self-expertise and instruction tuning, we aimed to distill two key attributes: the ability to follow instructions and legal domain knowledge. the model showed proficiency in generating responses based on legal statutes and case law, adhering to the logical framework of legal reasoning, as evidenced by high fluency scores. however, as the model learn the legal domain knowledge indirectly, it was prone to more frequent errors. prominent errors included hallucinations, where the model fluently provided responses based on inaccurate legal references. this suggests that while self-expertise has addressed knowledge issues compared to self-instruct, there are still aspects that remain to be resolved. we anticipate a combination of accurate knowledge acquisition methods with self-expertise to further enhance the precision in legal domain knowledge. we propose this potential improvement as a subject for future work. ## conclusion in this study, we propose self-expertise for automatically generating instruction dataset in specialized domain areas. our proposed method extracts knowledge from outputs of a seed dataset and uses this as a basis for generating instructions. this significantly reduces hallucination in instruction dataset creation. to demonstrate the effectiveness of the our augmentation method, we trained lx-pert with an augmented dataset on llama-2 7b and compared it with other baselines. it surpasses the performance of 7b models and gpt-3.5-turbo in terms of gpt-4 evaluation and human evaluation. we believe that this methodology can be extended and used for creating instruction datasets not only in the legal domain but also in other specialized knowledge domains. therefore, we look forward to the utilization of this augmentation pipeline when training an sllm specialized for knowledgeintensive tasks.