Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics (2025)

Abstract: Reinforcement learning with evaluation metrics as rewards is widely used to enhance specific capabilities of language models. However, for tasks such as factually consistent summarisation, existing metrics remain underdeveloped, limiting their effectiveness as signals for shaping model behaviour.While individual factuality metrics are unreliable, their combination can more effectively capture diverse factual errors. We leverage this insight to introduce an automated training pipeline that improves factual consistency in summaries by aggregating scores from different weak metrics. Our approach avoids the need for complex reward shaping by mapping scores to preferences and filtering out cases with high disagreement between metrics. For each source document, we generate lexically similar summary pairs by varying decoding strategies, enabling the model to learn from factual differences caused by subtle lexical differences. This approach constructs a high-quality preference dataset using only source documents.Experiments demonstrate consistent factuality gains across models, ranging from early encoder-decoder architectures to modern large language models, with smaller models reaching comparable factuality to larger ones.

## introduction cutting-edge language models have demonstrated impressive capabilities in generating fluent and coherent responses to a wide range of prompts. however, maintaining faithfulness and factual consistency remains a persistent challenge, particularly in tasks like summarisation. despite their surface plausibility, model-generated summaries often contain factual inconsistencies or hallucinated details @xcite . recent research has tried to mitigate this issue by incorporating reinforcement learning (rl) to guide models towards more factually consistent outputs. a critical obstacle lies in designing effective reward signals that can reliably capture and quantify factuality. many existing automatic evaluation metrics @xcite @xcite have been adopted as reward signals for rl. however, even state-ofthe-art metrics struggle with subtle inconsistencies and may penalise factually accurate outputs @xcite . furthermore, using a single metric directly as a scalar reward for rl can lead to unstable training, as explored by @xcite . the training process is influenced by the metric's reliability and the distribution of reward scores, but the distributions of existing evaluation metrics are not well-studied. although combining metrics can broaden error detection coverage @xcite , applying metric combination in rl often requires manual weighting of sub-rewards @xcite @xcite , thus the use of rl is impeded by the complexity of reward shaping. an alternative is reinforcement learning from human feedback @xcite , which uses human-annotated preferences and has proven effective at aligning large language models (llms) with broad human values. however, recent work @xcite shows that existing rlhf datasets often overlook factuality, despite instructions to annotators to account for it. annotator judgments integrate various considerations, such as trade-offs among properties, individual biases, and occasional misunderstandings, so the resulting overall preferences can fail to reliably capture factuality. therefore, while constructing a factuality-focused preference dataset is crucial, doing so requires substantial resources and expertise, making scalability a major concern. to overcome these barriers, this paper proposes a fully automated training pipeline that improves factual consistency in summarisation without rely-summary1: four individuals, including three 18 -year-olds and a 24-year-old, have been charged with hate crimes and aggravated kidnapping and battery in connection with the brutal beating of an 18-year-old white victim with schizophrenia and attention deficit disorder. summary2: four chicago suspects, including an 18 -year-old schizophrenia sufferer, have denying bail after their charges with multiple hate crimes and aggravated kidnapping in a horrific beating from which has raised an alarming and devastating $51,000 in aid through an open online ## conclusion we introduce a novel automatic training pipeline for improving the factual consistency of summarisers. our approach can be generalised over different model architectures and scales. it requires only source documents, utilising multiple factuality evaluation metrics to score the summary and obtain labels for preference optimisation. the experimental results suggest that our approach outperforms baselines and boosts the factuality performance of smaller models to a level comparable to llms, revealing the effectiveness of preference learning over similar summary pairs.