Regularized Contrastive Decoding with Hard Negative Samples for LLM Hallucination Mitigation (2025)

Abstract: Large language models are prone to generate hallucinations, which can undermine their reliability in high-stakes applications. Some works on LLM hallucination mitigation use the model’s internal signals to contrast different output during inference stage. However, these works often focus on simple forms of hallucinations, and struggle to effectively mitigate hallucinations. To address the issue, this paper exploits hard negative samples to construct a factually weaker model for improving contrastive decoding. We propose a new inference-time method, Regularized Contrastive Decoding (RCD), to capture correct hallucination signals for mitigating hallucinations in LLMs. RCD learns more diverse hallucination patterns via adversarial-aware fine-tuning and mitigates hallucinations via contrastive decoding. Experiments on four hallucination benchmarks demonstrate that our method achieves better LLM hallucination mitigation performance. Further analysis shows RCD generalizes well across different model sizes, task formats, perturbation methods and training data sizes.

## introduction large language models (llms) have demonstrated substantial progress in a wide range of natural language processing (nlp) tasks @xcite . however, despite these achievements, llms often produce hallucinations outputs that factually incorrect or unfaithful to the provided context @xcite . these hallucinations pose significant risks, particularly in high-stakes domains such as legal consultation, medical advice, and specialized technical support. various strategies have been pursued to mitigate llm hallucination. some works leverage external * corresponding author. in which year was the 'boxing day tsunami' in the indian ocean? knowledge bases via retrieval augmented generation to improve factuality of model outputs @xcite . although effective in many settings, these methods usually require additional infrastructure and are sensitive to retrieval errors. other works rely on the model's internal signals to contrast different outputs without external retrieval, offering simplicity and ease of deployment @xcite @xcite . however, such methods often struggle to provide subtle hallucinations that are semantically close to the truth, resulting in suboptimal hallucination mitigation in llms. to provide more accurate hallucination signals, some studies use existing hallucination data to learn implicit representations @xcite . however, they focus on explicit and easily recognizable hallucinations, leading models to fit the specific patterns and biases of the limited training data. as a result, by providing incorrect hallucination sig-nals during contrastive decoding, these methods fail to effectively mitigate hallucinations, particularly when facing subtle cases in more complex scenarios. in this paper, we propose a new inference-time method, regularized contrastive decoding (rcd), to mitigate hallucinations in llms by contrasting against hard negative samples. inspired by @xcite , rcd generates hard negative samples via adversarial training for better contrastive decoding. specifically, first, we introduce an adversarialaware fine-tuning with lora @xcite to construct a factually weaker llm by inducing more diverse hallucination patterns. we apply fast gradient method (fgm) @xcite to generate adversarial perturbations. then we put perturbations on the embedding layer to generate hard negative samples with a min-max training strategy. as shown in figure 1 , these generated hard negative samples allow the weak llm to capture more precise hallucination patterns. then, we perform contrastive decoding with the factually weaker model. this enables more diverse and accurate hallucination signals, yielding outputs that are more factual and reliable. we conduct experiments on four public hallucination benchmarks, i.e. truthfulqa @xcite , factor @xcite , triviaqa @xcite and natural questions (nq) @xcite . experimental results demonstrate that rcd yields consistent gains across tasks. for instance, rcd achieves +19.75 absolute improvements on truth-fulqa mc2 and +12.71 accuracy scores on fac-tor expert. further analysis shows that rcd preserves the base model's performance on mmlu and arc-challenge. latency measurements confirm that it introduces only negligible overhead compared to standard contrastive decoding. moreover, rcd is compatible with different adversarial training strategies, scales well with model size across model sizes, and consistently achieves effective hallucination mitigation across different training data sizes. our contributions are summarized as follows: 1) we introduce hard negative samples to construct a factually weaker model for improving contrastive decoding. 2) we propose a new inference-time method rcd, using diverse hallucination signals to enhance contrastive decoding for hallucination mitigation in llms. 3) experiments on four hallucination datasets demonstrate that rcd consis-tently achieves better hallucination mitigation performance. rcd also generalizes well across different model sizes, task formats, perturbation methods and training data sizes. ## regularized contrastive decoding (rcd) consider a standard text generation setting where an llm receives an input sequence @xmath0, x 2 , . . . , x l ) and generates an output sequence @xmath1, y 2 , . . . , y t ). without additional constraints, the llm may produce hallucinations, which are tokens or phrases unsupported by factual evidence. these hallucinations degrade the trustworthiness and reliability of the generated text. as shown in figure 2 , we propose regularized contrastive decoding (rcd) to improve hallucination mitigation by performing contrastive decoding between a strong model and an adversarially trained weaker model. inspired by @xcite , which generate worst-case samples to constrain contrastive representations, rcd generates hard negative samples via adversarial training for better contrastive decoding. ## conclusion we present regularized contrastive decoding (rcd), a new inference-time method that leverages hard negative samples to enhance contrastive decoding and achieve more effective hallucination mitigation. rcd learns diverse hallucination patterns to enhance the weaker model through adversarial-aware fine-tuning and employs contrastive decoding to mitigate hallucinations effectively. experiments on four public hallucination benchmarks demonstrate that rcd consistently obtain better hallucination mitigation performance. experiments also verified the effectiveness of rcd across different model sizes, task formats, perturbation methods and training data size.