Self-training Large Language Models through Knowledge Detection (2024)

Abstract: Large language models (LLMs) often necessitate extensive labeled datasets and training compute to achieve impressive performance across downstream tasks. This paper explores a self-training paradigm, where the LLM autonomously curates its own labels and selectively trains on unknown data samples identified through a reference-free consistency method. Empirical evaluations demonstrate significant improvements in reducing hallucination in generation across multiple subjects. Furthermore, the selective training framework mitigates catastrophic forgetting in out-of-distribution benchmarks, addressing a critical limitation in training LLMs. Our findings suggest that such an approach can substantially reduce the dependency on large labeled datasets, paving the way for more scalable and cost-effective language model training.

## introduction large language models (llms) have revolutionized natural language processing (nlp), enabling remarkable performance across various downstream tasks @xcite . however, their development is heavily reliant on vast amounts of labeled data and significant computational resources, which are not always readily accessible @xcite . self-learning is an applicable field that can tackle such limitations and enables a lowresource training environment. however, llms are known to hallucinate @xcite due to the inherent biases, noise in their pre-training dataset, or just lack of data. this makes it challenging to apply self-learning to continuously improve the knowledge of the model. another key problem in llm fine-tuning is catastrophic forgetting @xcite . this phenomenon occurs when the model learns new information in one domain but simultaneously suffers from a degradation of knowledge in previously 1 https://github.com/wj210/self-training-llm acquired areas. a naïve solution is to exploit larger data mixing new and old knowledge, which may not be feasible for domains with limited resources. alternative approaches, such as continual learning @xcite and inferenceonly correction @xcite offer potential solutions. however, these methods face other limitations such as reduced efficiency in learning new knowledge and scalability towards large domains. to address the aforementioned limitations, this paper explores a self-training paradigm where the llm autonomously curates its own labels and performs selective training on samples filtered using a new knowledge detection. this measure identifies instances that are annotated as "unknown", indicating the model's low confidence in providing accurate answers @xcite . this filtering step is specifically used to curate a preference dataset to perform knowledge correction via direct preference optimization (dpo) @xcite . the rationale behind performing the selection step is twofold. firstly, this allows for a larger distance between the preferred and dispreferred sample, thereby reducing noise in the training. this helps to prevent degeneration, a common issue observed when implementing dpo @xcite . secondly, training exclusively on samples related to lack of knowledge is resource-efficient and aids in retaining previously learned information. our results demonstrate that the proposed framework enhances factual accuracy in answering questions pertaining to a specified knowledge source. additionally, training on the selected samples not only preserves but, in some instances, improves performance on out-of-distribution benchmarks. comparative analyses with baseline approaches, which demand higher computational resources, reveal that our approach outperforms these baselines. ## self-training this section introduces the details of our selftraining framework, broken down into four sequential steps: instruction generation, sft stage, preference labeling and knowledge filtering. as a start, we assume access to a knowledge source as the main source of material to perform both training and truthfulness evaluation. the full illustration is shown in figure 1 . a key benefit of our framework is that it does not require significant human efforts besides a few manually crafted instruction-answer examples for in-context generation. ## experiments through the following experiments, we would like to answer three research questions: rq1: are llms capable of performing selftraining to improve truthfulness in responses? rq2: how does conducting selective training improve truthfulness in llms and what are the effects of forgetting on out-ofdistribution tasks? rq3: how sensitive is the knowledge filtering threshold, τ k with respect to mitigating hallucinations? test: the primary test dataset comprises the held-out questions curated from the target topics discussed in sec. 3.1, with 10 documents for each of the 10 topics, we refer to this as wiki-test. we construct 2 questions from each document, resulting in a total of 200 questions, generated using gpt-4 @xcite . we manually check the questions to ensure they are aligned with the documents. we also conducted experiments on the open llm leaderboard foot_1 consisting of various nlp benchmarks that are likely not to be directly included in the model's training set. the purpose of these evaluations is to detect signs of forgetting across tasks such as commonsense reasoning and general knowledge understanding when the model is fine-tuned on data of different distributions. ## results to compare different approaches with pairwise evaluation, we set the sft-ed model, g sf t as the baseline and compare all approaches against it. we run the evaluation twice for each instance, concluding with a tie if both evaluations disagree, similar to @xcite . ## conclusion in this work, we present a cost-effective approach to guiding llms to perform self-training on their own output. we develop a framework that minimizes human intervention and demonstrates that llms can self-correct errors through preferencetuning. specifically, our framework facilitates the creation of a high-quality preference dataset by excluding low signal-to-noise ratio samples using a knowledge detection technique. our experiments illustrate the dual benefits of our approach: enhancing the truthfulness of llms and promoting knowledge retention post-training. moreover, selftraining offers significant incentives such as maintaining data privacy, which is crucial for organizations hesitant to expose sensitive information to third-party platforms for dataset generation. this work opens multiple avenues for future research. given that the context is built upon publicly accessible material that may have been exposed to the model during pre-training, an intriguing direction would be to investigate its impact on specialized domains such as healthcare reports or financial statements, where human-labeled data is often scarce and private. additionally, our current results are based on a single iteration. future work could explore the potential for continual improvement by augmenting the preference dataset with new context through successive iterations.