H oward U niversity - AI 4 PC at S em E val-2025 Task 3: Logit-based Supervised Token Classification for Multilingual Hallucination Span Identification Using XGBOD (2025)

Abstract: This paper describes our system for SemEval-2025 Task 3, Mu-SHROOM, which focuses on detecting hallucination spans in multilingual LLM outputs. We reframe hallucination detection as a point-wise anomaly detection problem by treating logits as time-series data. Our approach extracts features from token-level logits, addresses class imbalance with SMOTE, and trains an XGBOD model for probabilistic character-level predictions. Our system, which relies solely on information derived from the logits and token offsets (using pretrained tokenizers), achieves competitive intersection-over-union (IoU) and correlation scores on the validation and test set.

## introduction semeval-2025 task 3, multilingual shared-task on hallucinations and related observable overgeneration mistakes @xcite addresses the critical challenge of detecting hallucinations in instruction-tuned large language model (llm) outputs. the challenge of hallucination extends beyond text-based llms to multimodal large language models (mllms) as well, posing significant obstacles to their real-world applications @xcite . this task is crucial for ensuring the reliability and trustworthiness of llms in real-world applications, especially in multilingual contexts. mu-shroom encompasses 14 languages: arabic, basque, catalan, chinese, czech, english, farsi, finnish, french, german, hindi, italian, spanish, and swedish, reflecting the growing need for robust multilingual llm evaluation @xcite . our system tackles this span detection task by framing it as a point-wise anomaly detection problem. we hypothesize that hallucinated text spans exhibit anomalous patterns in the llm's output logits compared to factual or consistent text. inspired by recent work demonstrating the potential of large language models for time series anomaly detection @xcite , we leverage the xgbod (extreme gradient boosting for outlier detection) algorithm @xcite , trained on features extracted directly from the llm's logit sequences, to identify these anomalous points indicative of hallucinations. by participating in mu-shroom, we discovered that a relatively simple, data-driven anomaly detection approach can achieve effective hallucination span detection across diverse languages, relying solely on model logits without prompts, text, tokens, or retrieval-augmented generation (rag). ## background recent work has also explored the use of llms directly for time series anomaly detection. @xcite proposed llmad, a framework that uses llms for few-shot anomaly detection in time series, achieving both high accuracy and interpretability. their work, while focused on general time series data, further motivates our exploration of anomaly detection techniques for hallucination detection in llm text output, particularly by leveraging the llm's own logit representations. the survey by @xcite provides comprehensive overviews of various hallucination detection techniques. the field of time series anomaly detection itself is a well-established area, with extensive research into various methodologies, as highlighted in comprehensive surveys by @xcite . these surveys cover a wide range of techniques, including deep learning approaches, and discuss applications across diverse domains. anomaly detection techniques have been successfully applied to various time-series data domains, such as system log analysis @xcite . the task input consists of: the expected output from participating systems is, for each character in the model_output_text, a probability indicating whether it is part of a hallucination span. the dataset is split into sample, validation, unlabeled train, unlabeled test, and labeled test sets. we utilized the validation set for training our anomaly detection model and the unlabeled test set for evaluation. the dataset covers 14 languages and utilizes outputs from various public-weight llms. our submission focused on span detection across all 14 languages using a single model. ## system overview our system employs a three-stage process: feature extraction from logits, anomaly detection using xgbod and prediction of anomaly scores for each token. ## results as shown in ## conclusion in this paper, we presented our system for semeval-2025 task 3: mu-shroom. our approach reframes multilingual hallucination span detection as a point-wise anomaly detection problem on llm output logits, utilizing the xgbod algorithm @xcite . experimental results demonstrate the effectiveness of this simple yet powerful approach, achieving competitive performance across 14 diverse languages. while our system shows promising results, we acknowledge several limitations and directions for future work. a key limitation is that our point-wise anomaly detection with xgbod, while effective, does not explicitly model the temporal dependencies within the logit sequence. furthermore, accurately calculating token offsets proved challenging across diverse models due to varying tokenizer support, sometimes necessitating reliance on less precise string parsing approaches. future research will explore directly learning from these temporal dependencies by treating logit sequences as time series, potentially using sequence models within frameworks like ludwig. we are also actively investigating integrating retrieval-augmented generation (rag) and textual information to provide richer context for hallucination detection. this includes exploring a multimodal approach to leverage diverse data sources. addressing the inherent data scarcity and the challenges of obtaining consistent human labels across multiple languages and models remains a crucial long-term goal for advancing research in this domain. we believe that further exploration of these directions will lead to more robust and accurate multilingual hallucination detection systems.