Not all Hallucinations are Good to Throw Away When it Comes to Legal Abstractive Summarization (2025)

Abstract: Automatic summarization of legal documents requires a thorough understanding of their specificities, mainly with respect to the vocabulary used by legal experts. Indeed, the latter rely heavily on their external knowledge when writing summaries, in order to contextualize the main entities of the source document. This leads to reference summaries containing many abstractions, that sota models struggle to generate. In this paper, we propose an entity-driven approach aiming at learning the model to generate factual hallucinations, as close as possible to the abstractions of the reference summaries. We evaluated our approach on two different datasets, with legal documents in English and French. Results show that our approach allows to reduce non-factual hallucinations and maximize both summary coverage and factual hallucinations at entity-level. Moreover, the overall quality of summaries is also improved, showing that guiding summarization with entities is a valuable solution for legal documents summarization.

## introduction the extensive use of large language models (llms) for summarization tasks brings new challenges. one of the most important is related to the llms' tendency to generate hallucinations, i.e., texts that give the impression of being fluid and natural, despite their lack of fidelity or nonsensical nature @xcite . hallucinations can be non-factual or factual. in this last case, a source of knowledge, such as an expert, an ontology or a knowledge base, can indeed certify their veracity @xcite . most of the current works on hallucinations aim at deleting, reducing or correcting non-factual hallucinations through dedicated models @xcite @xcite or post-processing approaches @xcite @xcite . some other works try to control factual hallucinations through the use of external knowledge, while keeping low the number of non-factual hallucinations @xcite @xcite . this last line of research is particularly of interest while considering specific domains such as the legal field. indeed, summaries of legal documents written by experts often contain domain-specific knowledge and references to previous laws that help the readers to fully grasp the content. this is illustrated in figure 1 through an example of legal document with its associated reference summary, extracted from the eur-lex-sum dataset foot_0 . it can be observed that both the document and the reference summary include a large number of entities of interest (in bold). moreover, the reference summary mentions laws regulation 2019/979, regulation 2020/1273, and regulation 2020/1272 (colored in dark and light blue on figure 1 ) that do not appear in the document. such entities have been added to the summary by experts in order to track all the amendments made to the regulation 2019/980 presented in the document. they are named abstractive entities, in opposition to faithful entities, i.e., entities present in the source document @xcite (in green in the reference summary). abstractive entities, as they are written by experts, are de facto factual. in this paper, we focus on abstractive summarization of legal documents, with the main objective to produce enriched, useful, and factual summaries which will lighten the task of information monitoring by legal experts. to do so, we aim at controlling factual hallucinations by knowledge arisen from the source dataset. in other words, we propose an approach which aims at fulfilling a two-objectives task: (1) retrieve factual hallucinations from the dataset; and (2) keep very low the number of nonfactual hallucinations. figure 1 : example of a eur-lex-sum legal document with its reference summary (excerpts) produced by experts, as well as an example of generated summary. entities of interest are marked in bold. in the reference summary, faithful entities are colored in green. abstractive entities of the reference summary are colored in dark blue when present in the dataset (in-dataset abstractions) whereas the light blue one is an out-dataset abstraction. in the generated summary, faithful entities are also colored in green. the entities colored in red are non-factual hallucinations whereas the yellow one is a factual hallucination. our work focuses on hallucinations and abstractions at entity level for two main reasons: (i) entities are more striking to human readers @xcite , and (ii) the legal domain is overflowing with entities of interest @xcite . first, we retrieve entities from the source dataset that closely match the set of abstractive entities of the reference summary. then, we introduce various approaches for injecting the retrieved entities during the training of the model, in order to control the generated entities (faithful and hallucinated ones) at inference. to the best of our knowledge, this is the first approach aiming at controlling factual hallucinations using only the original dataset as a source of knowledge. to evaluate our approach, we conducted various experiments on two different legal corpora in english and french. we aimed at answering the following questions: ## legal summary specificities our aim here is to study the characteristics of reference summaries in terms of entities of interest as well as abstractions. to this end, we analyzed two legal datasets for summarization: eur-lex-sum and a private corpus of french legal news.