Knowledge Acquisition through Continued Pretraining is Difficult: A Case Study on r/ A sk H istorians (2024)

Abstract: Powerful LLMs like ChatGPT are adopted rapidly for a wide array of tasks, but their limitations in domain-specific areas become apparent, particularly when prompted to recite facts. This is critical especially for knowledge workers, who are adopting LLM-based tools rapidly.While there are various techniques that can help ingest knowledge into LLMs such as instruction tuning and alignment, most have disadvantages. We examine the impact of prominent training techniques on LLMs’ knowledge accuracy using a knowledge-dense dataset that we curate from r/AskHistorians, a rich source of historical knowledge. We evaluate the impact of different models sizes from 1.3B to 7B parameters and other factors such as LoRA adapters, quantization, overfitting, and the inclusion of Reddit data in pretraining.In addition, we measure linguistic metrics and human and LLM-based preference. Our results suggest that pretraining and model size have a much stronger effect on knowledge accuracy than continued pretraining – unless the model is overfit to the tested knowledge.Fine-tuning on our Reddit dataset introduces less complex, but slightly more toxic language. Our study explores the challenges of injecting domain-specific datasets into LLMs and has implications for practitioners, e.g., when LLMs are to be fine-tuned with a company’s datasets.

## introduction large language models (llms) have evolved far beyond mere natural language processing tools and are now widely used by knowledge workers seeking answers to knowledge-related questions. however, while these models incorporate a vast set of world knowledge due to their pretraining on trillions of tokens, they still often lack niche domainspecific knowledge, which can manifest in hallucinations or unspecific responses @xcite . in addition, an llm's knowledge may need to be updated from time to time @xcite . these issues are especially critical for models deployed in professional settings to assist knowledge workers in performing knowledge-intensive tasks in particular domains. there are various ways how one can try to ingest knowledge into llms, but each has its disadvantages: unsupervised pretraining enables llms to learn immense amounts of knowledge, but without long-tail details @xcite . supervised fine-tuning (sft, or instruction tuning) can be used to expose the model to new knowledge when learning a new task, but niche facts do not seem to "stick" @xcite and finetuning without the original training data can lead to catastrophic forgetting @xcite ; alignment using techniques such as reinforcement learning from human feedback (rlhf; e.g., as used in @xcite or direct preference optimization (dpo; @xcite ) can greatly improve the quality of generated texts and introduce safety mechanisms, but requires costly training due to very small learning rates for highly nuanced model adjustment. retrieval augmented generation (rag; @xcite appears to be a promising workaround that avoids fine-tuning, but requires a more complex architectural setup and a greater number of prompts and tokens to operate, causing higher usage costs, while the result quality is highly dependent on the information stored in its database. we investigate this area of research using a large dataset from r/askhistorians, a strictly moderated online community on reddit, that contains questions and long-form answers on diverse historical topics, often discussing very specific historical facts. as reddit users can up-or downvote posts and comments, the dataset provides inherent human feedback that can be leveraged for aligning llms with dpo. given that social media datasets often pose challenges with regard to issues such as data quality and toxicity, we exercise special care to curate a high-quality dataset. furthermore, we assess the impact of different model sizes (ranging from 7b to 1.3b parameters), the usage of lora adapters and quantization, and overfitting to the knowledge dataset. we present an approach to measure the knowledge accuracy of the models by manually creating a knowledge filling dataset. in addition, we conduct human and llm-based evaluation, and consider more traditional nlp metrics such as text complexity, reading time, and toxicity. the main purpose of this work is to demonstrate how one can proceed when attempting to inject specific knowledge into llms and evaluate its success. our code is publicly available on github foot_0 . ## methodology an overview of our technical setup is shown in figure 1 : we process and curate a preference dataset from the raw r/askhistorians data, utilize it for model training using sft (phi-1.5 and zephyr-7b-beta) and dpo (zephyr-7b-beta) and evaluate using different approaches, including gpt-4-turbo as llm judge, and mistral-7b-v0.1 and pythia-1.4b for baseline comparisons. ## limitations in some of our experiments, there is a risk of testset contamination, as the underlying training data is not transparently declared for all tested models (e.g., mistral-7b-v0.1) -it is possible that these models may have seen parts of our test dataset during their pretraining when using datasets like commoncrawl (common crawl, 2024) and the pile @xcite . we mitigate this by testing various model variants, including phi-1.5, which certainly has not seen reddit data in pretraining. also, the fact that zephyr has likely seen reddit data provides additional insights, as the decreased knowledge accuracy after fine-tuning and alignment potentially indicates reduced ability to generalise. our knowledge filling dataset for evaluation has a limited size, as its creation is highly timeconsuming and cannot be outsourced or automated easily, due to requiring the annotator to understand the contents of the annotated text correctly. despite meticulous curation, the dataset may inadvertently contain factually inaccurate statements. in addition, the setup as a cloze test leads to ambiguity: for instance, when prompted with "which world war ended in 1945?", the answer can either be "ww2" or "world war 2"; or specific dates may appear in different formats. this is mitigated in our study through detailed manual verification and would benefit from automation in future work. we employ additional evaluation techniques to provide diverse and more robust results. the number of conducted experiments and trained model variants was limited by our access to shared computational resources, which is why we were not able to train and evaluate all possible combinations of model variants. therefore, we have focused on providing a sufficient number of experiments to investigate the most interesting questions stated in the motivation of this paper. ## conclusion in this work, we show that there are various challenges when trying to inject knowledge into a llm by using common techniques like sft and dpo, and present an approach for evaluating the knowledge accuracy and stylistic quality of trained llms from various perspectives. while state-of-the-art llms like zephyr-7b-beta already generate high quality texts out-of-the-box (due to their training on carefully curated data) and tend to deteriorate when fine-tuned on domain-specific texts, conducting further fine-tuning may still be necessary for practitioners in order to adjust the models to their specific use case, e.g., company datasets. our approach is intended to inspire practitioners to conduct comparable experiments and evaluate their specific llms' knowledge accuracy, as the techniques that we apply are generalizable and transferable to other domains that require niche or fact-related knowledge. for future work, there are various open questions, such as identifying more powerful ways to inject knowledge into llms and facilitating the creation of similar knowledge benchmarks at a larger scale.