LORE : Continual Logit Rewriting Fosters Faithful Generation (2025)

Abstract: As autonomous agents and assistants, large language models (LLMs) often struggle with “hallucinations.” Fundamentally, the problem is one of prioritization and balance: the LLM needs to understand or infer when it needs to be creative and balance that with its need to be accurate. Most efforts focus on either updating intrinsic knowledge via targeted post-training or by adding external knowledge sources which the LLM can reference neurosymbolically (e.g., via retrieval-augmented generation). However, these all eventually rely on the LLM’s implicit reasoning ability during generation, still allowing for these random hallucinations despite high-quality training examples and references. Using aspect-oriented summarization as a case study, we propose **LOgit REwriting**(**LORE**), a new controlled generation paradigm which can simultaneously be faithful to external knowledge and to the LLM’s intentions. LORE works by adding a rewriting module at left-to-right inference time, continuously reflecting on the newest prediction and trying to find a replacement that is more faithful to the source document. Then, it merges the logits of the replacement with those of the original prediction to generate the next token. We created a new long-context aspect-oriented summarization dataset, **SLPAspect**, and find that LORE generates 5.8% better summaries compared to the LLM without LORE-rewriting. All code and data from this paper will be available on GitHub after the anonymity period.

## introduction the most widely used llms are capable of generating extremely coherent and reasonable text, often surpassing the efficiency and eloquency of humans @xcite @xcite . however, for domains like education and medicine, where providing the right or wrong treatment or applying an effective or ineffective intervention can have long-lasting effects on the student or patient, hallucinations often undermine the efficacy of llms, eroding the trust of users @xcite . due to the infinite input space of prompts and their effects on the generated output, developers continuously update their prompts to reduce perceived hallucinations in document understanding or summarization. however, it is especially in these cases where the dangers of hallucinations prove to be most harmful: when the "95%" of a generation that a human can easily verify looks good, it's simple to take the remaining 5% at face value. a human may tune a prompt to seemingly eliminate hallucinations foot_1 , but without a mechanism to constantly avoid them at inference time, this can lead to catastrophic failures when deployed. these risks are widely pertinent in both the simplest case of prompting an llm to operate on an explicit source document and in larger systems where retrieval-augmented generation (rag) is used @xcite . to reduce hallucinations in text generated based on source documents, we propose logit rewriting (lore), a framework for controlling the generation of an llm and ensuring that the output text is faithful to both the source document and the llm's creativity and language modeling ability. lore does this by using a natural language inference (nli) model to continuously ground the generated output to the source document. then, in cases of hallucination, lore changes the offending output into text that can be grounded back to the source. unlike many other frameworks, lore allows for rewriting across any token in the vocabulary of the nli model, as opposed to only reranking options retained in the current generation beam. to evaluate the efficacy of lore, we also introduce a new dataset for aspect-oriented summarization called slpaspect focused on the speech-language pathology domain foot_2 . the speech-language pathology domain is a quintessential usecase where llms need to be highly accurate and hallucination-free with respect to their source to reduce risk of harm (we will discuss this further in §2). to summarize, our primary contributions are: ## task & data aspect-oriented summarization @xcite is a task where faithfulness is particularly important, as each document may have multiple different aspects for which different parts of the source document are relevant. given an aspect (a small subtopic of interest) and a source document, we aim to generate a summary of the source document's information relating to only that aspect, which we will call an "aspect summary." existing aspect-oriented summarization benchmarks, such as aspectnews @xcite , massw @xcite , and aclsum @xcite focus on domains with relatively low risk (news and nlp scholarly documents respectively) compared to higher risk domains such as the medical and education domains. as with other medical practitioners, speech-language pathologists (slps) are constantly trying new ways for evaluating and treating patients, often school students with speech and language disorders. however, the patient load for slps is extraordinarily high among medical professions, with the average slp reporting a caseload 25% larger than manageable and 76% of slps believing there to be barriers to maintaining manageable caseloads; of the 76%, a third identified the primary barrier being a shortage of slps (association, 2024). thus, most slps rarely have time to read the latest literature to find evidence-based diagnosis methods and interventions and have turned toward llms to reduce this burden by extracting information to summarize or synthesize scholarly articles @xcite . in these cases, the risk of harm is high, as llms are highly capable of producing reasonable-sounding aspect summaries, but if these contain hallucinations, the already timeconstrained slps may not realize this and end up applying interventions which are not supported by the literature. to help research in the intersection of nlp and speech-language pathology, as well as to highlight the difficulty of faithfulness in a domain where most texts seem logical but are only high quality when evidence-based, we introduce slpaspect as a new long context aspect-oriented summarization dataset. in this higher risk domain, hallucinations can have more consequences while simultaneously being more difficult to recognize. slpaspect is constructed by collecting articles from three journals: international journal of speech-language pathology (ijslp), language, speech, and hearing services in schools (lshss), and american journal of speech-language pathology (ajslp). these are three of the top journals published by the american speech-language-hearing association (asha), the leading organization on slp research and practices. in a subset of these articles (as this is not required by the journals), the original authors wrote "headline"-type abstracts summarizing key aspects of their articles (e.g., purpose, method, results, conclusion, etc.). so, to construct slpaspect, we collected all volumes of the associated journals and parsed their text to use these headline abstracts as ground-truth aspect summaries. a summary of dataset statistics is presented in tables 1 and 2 , with example ground-truth aspect summaries shown in appendix a.1. ## methodology architecturally, lore uses the concept of "rewriting" as a way to control the generation. by continuously grounding the output back to the source document as we perform left-to-right generation, we can have a fine-grained control over faithfulness to both the llm doing the generation as well as to the source document itself with built-in source (i) current generation o i (timestep t) input i rewriting module rewritten logits p compute grounding score g nli model find alt. logits to improve g calculate gradients for g improved logits m'(i) weighted average with faithfulness parameter α left-to-right generation legend generation steps including rewriting inputs to rewriting are partitioned during pre-current generation [i 1 , i 2 , ... , i t-1 ] newest token i t llm m (based on segment s) filter to only include output based on segment s: [i 5 , i 6 , ... , i t-1 ] embed tokens via nli model's embedding layer determine, for each v in the nli model's vocabulary, the probability that it improves g embeddings [e(i 5 ), e(i 6 ), ... , e(i t-1 ), e(i t )] nli model: calculate score @xmath0, i 6 , ... , i t-1 , i t ] compute gradient of g w.r.t. e(i t ) processing. thus, lore can be used plug-and-play with other methods across domains with different faithfulness measures for short or long contexts. ## related work aspect-oriented summarization. most aspectoriented summarization methods focus on shorter documents in specific formats, such as questionanswer forums @xcite or reviews @xcite @xcite . fewer methods focus on general long documents summarized for generic aspects. other approaches to aspect-oriented summarization include weak supervision @xcite , self-supervised pretraining @xcite , keyword-guided summarization @xcite , prompt engineering @xcite , and extractive summarization @xcite @xcite . in contrast to these methods, we utilize a rewriting mechanism to improve the natural generation capabilities of a pretrained large language model. faithfulness rewriting. faithfulness, which represents the factual consistency with the input text, plays a vital role in summarization. previous work has explored the role of post-editing @xcite , namely rewriting, to ensure such faithfulness. despite this, those post-editing techniques usually require training revision models @xcite @xcite . inspired by the success of logit editing in decoding phrases @xcite , we use the nli model as a signal to improve the consistency between source document segments and generation results. one direction close to our work is embedding steering @xcite @xcite . however, these efforts rely on external signals such as sentiment and toxicity to control the model's generation, while lore relies on the knowledge consistency between generation results and source document. separately, qiu et al. (2024) utilized general hypothesis verification models (of which nli is a subset) to rerank generations based on verification but can only operate over candidate sequences, while yu et al. (2023) used a similar classification modelbased signal but for post-training the pretrained language model itself. ## conclusions and future work in this paper, we introduced a paradigm called logit rewriting (lore) for generating faithful summarizations based on a canonical source document. we showed that lore's performance exceeds that of specially trained models and the latest closed-and open-source llms by utilizing "rewriting" during inference time to control the generation. we also introduced a new dataset called slpaspect for the aspect-oriented summarization task, which contains long-context data in the medical/education domains along with expert groundtruth aspect summaries. future directions for this line of work include applying lore to general-domain rag applications. furthermore, increasing the rewriting context (instead of only the current token) of lore and adding a deletion mechanism would help bridge the gap toward the seq2seq behavior of llm-based rewriting. also interesting would be rewriting using domain-specific classification models rather than only nli. despite our method's attempts to curtail hallucinations, application of lore is not enough to guarantee that generations will be accurate, safe, and hallucination free. we urge all readers to remain vigilant.