D oc ME dit: Towards Document-Level Model Editing (2025)

Abstract: Model editing aims to correct errors and outdated knowledge in the Large language models (LLMs) with minimal cost. Prior research has proposed a variety of datasets to assess the effectiveness of these model editing methods. However, most existing datasets only require models to output short phrases or sentences, overlooks the widespread existence of document level tasks in the real world, raising doubts about their practical usability. Aimed at addressing this limitation and promoting the application of model editing in real-world scenarios, we propose the task of document-level model editing. To tackle such challenges and enhance model capabilities in practical settings, we introduce DocMEdit, a dataset focused on document-level model editing, characterized by document-level inputs and outputs, extrapolative, and multiple facts within a single edit. We propose a series of evaluation metrics and experiments. The results show that the difficulties in document-level model editing pose challenges for existing model editing methods.

## introduction large language models (llms) have demonstrated exceptional performance across a wide range of fields and are widely applied in various practical scenarios @xcite @xcite . given their broad usage, it is crucial for llms to deliver accurate and reliable information. however, llms may still generate incorrect or outdated information due to the knowledge they store, which can be inaccurate @xcite . such inaccuracies can lead to serious consequences in critical domains, such as medical diagnoses and legal advice, highlighting the importance of methods to correct errors in llms. to address this issue without expensive retraining, model editing techniques have been proposed @xcite @xcite . to evaluate the effectiveness of model editing methods, previous researchers have proposed a range of datasets encompassing various tasks such as single-hop and multi-hop question answering, cloze tasks, and others @xcite @xcite . to verify whether a model knows specific facts, almost all of these datasets require the model to output a short phrase or sentence. as shown in figure 1 , prior datasets only demand shorter textual outputs from the model. however, in real-world scenarios, document-level tasks such as generating biographies, long chains of thought, or updating wikipedia documents are more common. moreover, the document-level model editing challenge for model editing methods lies in the need to extrapolate answers from facts, handle longer contexts, and deal with multiple facts within a single document. due to the lack of research on these challenges, the application of existing model editing methods to practical llms is limited. to promote the application of model editing in document-level tasks, we propose a new task: document-level model editing. document-level model editing requires both the input and output to be at the document level. additionally, the model cannot infer the answer from the given facts; it involves longer contexts and multiple facts within a single edit. aimed at addressing the lack of practical applicability in existing benchmarks and promoting model editing at the document level, we propose a novel benchmark: docmedit, a model editing dataset that contains document-level data. the input and output of docmedit are both at the document level. besides, the model cannot derive the answer solely from the facts to be edited; instead, it must combine multiple facts to be edited with existing knowledge to produce the updated document. unlike most previous research, our editing facts are not derived from triples but are directly extracted from unstructured data in wikipedia, which is more aligned with real-world model editing scenarios. docmedit contains 37,990 data items, with an average context length of 1,535.5 per item. it includes 105,652 editing facts, with an average of 2.78 facts per data item. additionally, we extracted triples from docmedit that share the same relations as those in wikidata to facilitate experiments with rag-based methods @xcite . to validate the effectiveness of existing model editing methods for document level model editing tasks, we propose a series of new metrics and conduct experiments on docmedit. we also further discuss how each challenge in document level model editing affects the performance of model editing methods. the conclusion demonstrates that existing methods have low accuracy while exhibit-ing strong side effects. further research confirms that factors such as document length, fact length, number of facts, and fact updates all impact the effectiveness of model editing methods. the main contributions of this paper are as follows: ## problem definition model editing aims to modify the knowledge contained within a model, changing the output related to the facts to be edited while keeping other outputs unchanged. based on previous research @xcite @xcite , we define document level model editing as follows: the document-level model editing task aims to modify the document-level output of a large language model based on a set of facts to be edited while keeping the parts unrelated to these facts unaffected. specifically, let the large language model be denoted as m, with the input and output of the model represented as x and y, respectively, i.e., @xmath0. the output y consists of multiple sentences, represented as @xmath1, where s i are the sentences of the original document. in the document level model editing, the original large language model m generates an output y based on the input document x, and then the model is edited using a set of facts to be edited @xmath2, resulting in the edited model m ′ , which generates a new output document y ′ = m ′ (x). the new document y ′ consists of the original sentences as well as newly added sentences supported by the facts to be edited. it is represented as y ′ = n @xmath3, where s i are the sentences consistent with the original text, and s f j are the new sentences supported by the facts to be edited. in this process, it is required that all sentences unrelated to the facts to be edited, s i , remain unchanged, while the newly added sentences,s f j , must be content supported by the facts f j . ## docmedit: a dataset dedicated to document level model editing to advance the practical application of document-level model editing, we introduce docmedit (document level model edit). the input and output in docmedit are document-level, with each document containing one or more facts to be edited. we also extract triples from each document and fact to facilitate the use of knowledge graph-based model editing methods. figure 2 : the construction process of docmedit. in the document change computation, we calculate the updates of documents in the wikipedia between two time points and retain those documents that exhibit entity updates. in the facts collection, based on the newly added entities within the documents, we identify the newly added sentences mentioning these entities and extract them as supporting facts. in the knowledge graph extraction, we extract structured knowledge graphs and impose constraints based on wikidata relations. (3) knowledge graph extraction. the detailed steps are as follows: ## experiment in this section, we present our main experiments. we introduce the edited models used in section 5.1, the baseline model editing methods in section 5.2, and the evaluation metrics in section 5.3. we analyze the results of our experiments in section 5.4. for more experimental details, please refer to appendix c. ## analysis in this section, we conduct a deeper analysis to examine the key differences between document-level model editing and traditional model editing, as well as the challenges posed by these differences. we contend that the main differences between document-level model editing and traditional model editing are: 1) the length of the input and facts to be edited is longer, and 2) multiple parallel facts to be edited are allowed within a single editing objective. to substantiate our conclusions, we design a series of research questions (rqs) and analyze their impact on editing effectiveness. for the first aspect, we propose the following research questions: rq1a: the impact of context length on editing. rq1b: the impact of fact length on editing. for the second aspect, we propose: rq2a: the impact of the number of edited facts per document on the results. rq2b: the impact of fact updates on model editing performance. according to the findings of the main experiments, entity-based metrics are more indicative of fact-based modifications. therefore, in the analysis, we use de to evaluate document-level rq1a and rq2b, and adopt ee to assess edit-level rq2a. for all rqs, we conduct experiments on llama2. ## error analysis we conducted an error analysis to identify the challenges in document level model editing. ## conclusion we introduce document level model editing, a model editing task that is more aligned with realworld applications. to advance research in this task, we present docmedit, a dataset focused on document-level model editing. experiments conducted on this dataset show that existing methods struggle with document-level model editing. further experiments indicate that the challenges of document level model editing stem from long contexts and the presence of multiple facts to edit within a single document, aspects that are overlooked by current methods. we hope that our research will propel the field of model editing forward and inspire further research in this area.