Instruction-tuned Language Models are Better Knowledge Learners (2024)

Abstract: In order for large language model (LLM)-based assistants to effectively adapt to evolving information needs, it must be possible to update their factual knowledge through continued training on new data. The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs. However, we find that LLMs trained with this recipe struggle to answer questions, even though the perplexity of documents is minimized. We found that QA pairs are generally straightforward, while documents are more complex, weaving many factual statements together in an intricate manner. Therefore, we hypothesize that it is beneficial to expose LLMs to QA pairs before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. Based on this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes on questions prior to training on documents. This contrasts with standard instruction-tuning, which learns how to extract knowledge after training on documents. Extensive experiments and ablation studies demonstrate that pre-instruction-tuning significantly enhances the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8%.

## introduction large language models (llms) store vast amounts of factual knowledge in their parameters through large-scale pre-training, and this knowledge can be used to answer various questions such as "where is the world's largest ice sheet located" @xcite @xcite @xcite . however, this factual knowledge is static, meaning that it can become outdated as the * majority of the work done during an internship at meta. world evolves, or prove insufficient when llms are used in specialized or private domains. to keep llms up-to-date, it is common to continue pre-training on new documents to store knowledge in parameters, which allows llms to effectively answer queries that require up-to-date information @xcite . a widely held view is that the factual knowledge stored in parameters can be elicited through prompting @xcite @xcite , and that instruction-tuning (also known as supervised finetuning or alignment) makes this elicitation more effective @xcite @xcite . in the first part of this paper ( § 4), we conduct extensive experiments using llama-2 @xcite to answer the following question: to what extent can we augment the knowledge stored in modern llms by continued pre-training on new documents, either with or without subsequent instruction-tuning? we find that, as we train llms repeatedly over documents to the extent that perplexity is minimized to one, the percentage of questions regarding those documents that llms answer correctly increases consistently to 27.6%. subsequent instruction-tuning further improves it to 30.3%, confirming that this widely used practice is useful to elicit more knowledge from llms. 1 however, the amount of elicited knowledge is still limited, even though the perplexity of documents is minimized, a phenomenon we refer to as the "perplexity curse". 2 in the second part of the paper ( § 5), we study methods to mitigate the perplexity curse by making llms more adept at absorbing knowledge from documents. @xcite presented an intriguing finding that training a randomly initialized transformer from scratch on a mix of biographies and related questions resulted in strong generalization to new questions. however, understanding the reasons behind this finding and exploring ways to practically apply it for absorbing knowledge from new documents requires further investigation. we found that question-answer (qa) pairs are generally straightforward and easily digestible, while documents tend to be more complex and cluttered, often weaving many factual statements together in a more intricate manner. therefore, we hypothesize that it is beneficial to deliberately expose llms to qa data before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. we refer to this as pre-instruction-tuning (pit) and conduct comprehensive experiments to benchmark different variations of this method. as shown in fig. 1 , our best-performing variation starts with training exclusively on qa pairs (e.g., "who handled the editing of oppenheimer") to grasp how knowledge is accessed. this is followed by training on a combination of these qa pairs and associated documents (e.g., "who handled the editing of oppenheimer" and a document about "oppenheimer"). in this phase, llms enhance their ability to absorb knowledge from information-dense documents, building upon the qa pairs that they have already mastered. to study continual knowledge acquisition, we build a dataset named wiki2023, which includes a collection of documents from wikipedia that are relevant to the year 2023. comprehensive experiments on wiki2023 demonstrate that after pit, llms exhibit an enhanced ability to absorb knowledge from new documents (e.g., a document about "barbie"). detailed ablation studies reveal that this ability primarily stems from prioritizing learning how to access knowledge over learning to encode knowledge from documents. overall, pit significantly outperforms the standard instruction-tuning approach ( § 5.1 and § 5.2), improving qa accuracies by 17.8% on llama-2 7b (30.3% 48.1%) and 16.3% on llama-2 70b (46.4% 62.7%). moreover, pit also enhances the ability to absorb knowledge from documents of a different domain, shedding light on the potential to scale this method up to a wider variety of documents and instructions for more robust generalization ( § 5.4). ## building a dataset to study continual knowledge acquisition to assess the ability of llms to learn knowledge from new documents, it is essential to use a document corpus with minimal overlap with the original pre-training corpus. this ensures that when an llm correctly answers questions, we can confidently attribute this capability to its learning from the new documents, rather than encountering similar questions in its original pre-training corpus. in this section, we describe a methodology for building such a corpus from wikipedia. ## how much knowledge can llms absorb via continued pre-training followed by instruction-tuning? factual knowledge stored in the parameters of llms can be accessed and applied to answering questions through prompting without additional training @xcite @xcite . with additional instruction-tuning (also known as supervised fine-tuning) on high-quality data @xcite , knowledge seems to be more effectively elicited from llms. however, when llms correctly answer a question, the source of the knowledge is unclear due to the diversity of the pre-training data. for instance, when answering the question "where is the world's largest ice sheet located", do llms derive their response by recalling and generalizing information from a seen document about the antarctic ice sheet, or do they merely repeat answers from similar questions encountered in the training data? this distinction is crucial, as the former scenario implies an ability to comprehend documents and effectively store knowledge within parameters in a way that can be elicited later, whereas the latter is mere rote memorization. several works have studied this problem and the predominant finding is that lms struggle to answer questions about documents they have been trained on @xcite . it is important to note, however, that these experiments were mainly conducted using relatively small lms such as bart, t5, or gpt-2 @xcite @xcite , using randomly initialized transformers (zhu and li, 2023a), or without instruction-tuning @xcite . this makes us wonder what are the actual limits of modern llms to absorb knowledge from new documents and answer questions about them using the standard continued pre-training followed by instruction-tuning recipe. in this section, we run extensive experiments using llama-2 7b and 70b on wiki2023-film to test their limits. ## improving llms in absorbing knowledge from documents the amount of knowledge elicited through the standard instruction-tuning is still limited, even though the perplexity of documents is minimized, a phenomenon we refer to as the "perplexity curse". our next question is how can we improve the ability of llms to absorb knowledge from documents to mitigate the perplexity curse. the main challenge is the gap between the way knowledge is presented in raw documents and how it is accessed through question-answering. we found that qa pairs are generally straightforward, while documents tend to be more complex and cluttered, weaving many factual statements together in a more intricate manner. using fig. 3 as an example, the answer to the question "who handled the editing of oppenheimer" is included in a sentence in the middle of the article "editing was handled by jennifer lame ...", which does not explicitly mention "oppenheimer". during training, llms must understand the context and deduce that "editing" refers to "the editing of oppenheimer" to effectively encode this knowledge in the parameters. zhu and li (2023a) studied this problem by training a randomly initialized gpt-2-like transformer from scratch on synthetic biographies and evalu-ated its ability to answer questions about the individuals. they found that training on a mix of biographies and questions related to half of those biographies led to strong generalization when answering questions about the remaining half of biographies, which resembles setting ➃ in fig. 4 . in contrast, training on biographies and qa pairs sequentially failed. however, the key contributor to the success remains uncertain because the data were blended together, and it is unclear how to apply this practically to absorb knowledge from new documents. inspired by our observation of the different difficulty levels between qa pairs and documents, and the finding from zhu and li (2023a), we hypothesize that it is beneficial to deliberately expose llms to instruction-tuning data before continued pre-training so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed. we refer to this as pre-instruction-tuning (pit) and study various implementations of pit prior to continued learning ( § 5.1), followed by detailed ablations identifying the keys contributor to performance ( § 5.2 and § 5.3), and finally assess how well pit performs across domains ( § 5.4). we adhere to the hyperparameters outlined in § 3.2 and perform pit for 3 epochs unless specified otherwise. ## conclusion we study the best way of continued training on new documents with the goal of later eliciting factual knowledge and propose pre-instruction-tuning that learns how knowledge is accessed via qa pairs prior to encoding knowledge from documents. extensive experiments and ablation studies demonstrate the superiority of pre-instruction-tuning versus standard instruction-tuning. future directions include scaling this method up to a broader range of documents and instructions for more robust generalization.