S ci BERT Meets Contrastive Learning: A Solution for Scientific Hallucination Detection (2025)

Abstract: As AI systems become more involved in scientific research, there is growing concern about the accuracy of their outputs. Tools powered by large language models can generate summaries and answers that appear well-formed, but sometimes include claims that are not actually supported by the cited references. In this paper, we focus on identifying these hallucinated claims. We propose a system built on SciBERT and contrastive learning to detect whether a scientific claim can be inferred from the referenced content. Our method was evaluated in the SciHal 2025 shared task, which includes both coarse and fine-grained hallucination labels. The results show that our model performs well on supported and clearly unsupported claims, but struggles with ambiguous or low-resource categories. These findings highlight both the promise and the limitations of current models in improving the trustworthiness of AI-generated scientific content.

## introduction large language models (llms) such as chat-gpt @xcite are increasingly used to support academic research by answering domain-specific questions and summarising scientific content. while their outputs are often fluent and persuasive, they may introduce statements that are not grounded in the source material -a phenomenon known as hallucination. detecting hallucinated claims is especially difficult in scientific domains, where language is highly specialised and reference documents are lengthy. in this work we present a contrastive-learning solution based on scibert @xcite for hallucination detection in scientific answers. our contributions are three-fold: (i) a systematic analysis of the scihal corpus that highlights the linguistic and structural challenges of the task, (ii) a multi-objective optimisation scheme that couples classification with two contrastive losses, and (iii) discussion and analysis of results and errors, including confusion matrix diagnostics, demonstrating the effectiveness of the proposed model. ## related work early studies of factual consistency focused on abstractive summarisation, where hallucinations degrade summary quality. @xcite showed that even state-of-the-art models hallucinate frequently, motivating automatic detection methods such as question answering (qa)-based factuality probes @xcite . with the advent of large language models (llms) like gpt-3, hallucinations have been documented in open-domain question answering @xcite and conversational agents @xcite . most approaches frame hallucination detection as either an entailment problem, requiring reference retrieval and contradiction detection, or a generation-probability anomaly task. contrastive objectives have proven effective at learning semantically meaningful representations from limited supervision @xcite . in factuality research, @xcite applied supervised contrastive loss to claim verification, achieving gains over cross-entropy-only training. yuan et al. (2022 ) employed information noise-contrastive estimation (infonce) to align biomedical entity mentions with definitions, improving downstream question answering performance. for hallucination mitigation, shi et al. (2023) used retrieval-augmented contrastive tuning to discourage unsupported generations, while deng et al. (2024) introduced dual-encoder contrastive pretraining to rank evidence passages. our work differs by combining two contrastive losses: triplet and infonce with a cross-entropy objective inside a scibert backbone, targeting both coarse and fine-grained hallucination labels in scientific texts. ## task description the hallucination detection for scientific content (scihal) shared task addresses a challenge in the use of generative ai-powered academic research assistants: the detection of hallucinated claims in automatically generated scientific answers. these hallucinations-claims unsupported by reliable sources-undermine the trustworthiness of ai-generated scientific content. the task is formulated as a multi-label classification problem, where participants are required to assess the factual consistency of claims generated in response to research-related questions. for each instance, participants are provided with: a question related to scientific research, a summarized answer produced by a generative ai system, an extracted claim from that answer, and the corresponding reference abstracts cited in support of the summary. participants must determine whether each claim is factually supported or hallucinatory based on the provided reference materials. the task is divided into two sub-tasks: coarse-grained hallucination detection, and fine-grained hallucination detection. ## conclusion we have introduced a contrastive-learning extension of scibert for detecting hallucinated claims in ai-generated scientific answers. jointly optimising cross-entropy with triplet and infonce losses yields consistent gains on both coarse-and finegrained settings of the scihal 2025 benchmark, outperforming an unweighted baseline and a purely cross-entropy model. the improvement is most pronounced for majority and medium-frequency labels, confirming that semantic alignment objectives complement token-level supervision. nonetheless, the model still struggles with the under-represented classes, indicating that re-weighting alone cannot fully offset data scarcity. future work could improve performance by model updates along three possible axes. first, coupling the encoder with a retrieval-compression module that distills each reference into a handful of salient sentences could help by thereby shortening inputs while preserving key evidence. second, we intend to introduce a curriculum that over-samples rare labels and structurally complex claims early in training, then relaxes the sampling schedule as the model stabilizes. third, we will examine whether parameter-efficient fine-tuning of substantially larger transformer backbones improves robustness, especially on the sparsest categories, without incurring prohibitive computational cost.