ANHALTEN : Cross-Lingual Transfer for G erman Token-Level Reference-Free Hallucination Detection (2024)

Abstract: Research on token-level reference-free hallucination detection has predominantly focused on English, primarily due to the scarcity of robust datasets in other languages. This has hindered systematic investigations into the effectiveness of cross-lingual transfer for this important NLP application. To address this gap, we introduce ANHALTEN, a new evaluation dataset that extends the English hallucination detection dataset to German. To the best of our knowledge, this is the first work that explores cross-lingual transfer for token-level reference-free hallucination detection. ANHALTEN contains gold annotations in German that are parallel (i.e., directly comparable to the original English instances). We benchmark several prominent cross-lingual transfer approaches, demonstrating that larger context length leads to better hallucination detection in German, even without succeeding context. Importantly, we show that the sample-efficient few-shot transfer is the most effective approach in most setups. This highlights the practical benefits of minimal annotation effort in the target language for reference-free hallucination detection. Aiming to catalyze future research on cross-lingual token-level reference-free hallucination detection, we make ANHALTEN publicly available: https://github.com/janekh24/anhalten

## introduction detecting hallucinations in large pretrained language models (e.g., @xcite is critical for ensuring their reliability in realworld applications. most existing hallucination detection benchmarks focus on reference-based tasks (e.g., summarization, machine translation, question answering) @xcite @xcite , comparing model generated text against provided references. however, reference-based hallucination detection is not appropriate for free-form text generation, where obtaining ground-truth references in real-time demands sufficient and accurate preceding retrieval step. to address these challenges, reference-free hallucination detection approaches have been introduced @xcite , focusing on identifying inconsistencies within the generated context itself to effectively detect hallucinations in real-time. besides, most research in hallucination detection has concentrated on sentence or passage-level @xcite @xcite , which is inadequate for real-time applications that require immediate feedback during text generation. fine-grained, tokenlevel reference-free hallucination detection benchmark is necessary for this purpose. however, research in this area has focused on english @xcite , primarily due to the lack of robust evaluation datasets in other languages. creating token-level hallucination detection datasets for new languages (from scratch or using machine translation) is significantly more expensive and timeconsuming than for most other nlp tasks, due to the need for accurate translation and adaptation of nuanced contexts and token-level annotations. the lack of multilingual evaluation benchmarks hinders the investigation of cross-lingual transfer approaches for token-level reference-free hallucination detection. in this work, we target this gap and introduce anhalten (german halucination detection), a new benchmark derived from the english tokenlevel reference-free hallucination detection dataset hades @xcite . anhalten is: (1) reliable -with complete texts and hallucination spans (i.e., labels) manually translated, and (2) parallelthe same set of texts and labels have been translated to german, enabling direct comparison of multilingual models and cross-lingual transfer approaches. we then use anhalten to benchmark a range of cross-lingual transfer approaches and simulate the real-world applications in multiple setups. our hades anhalten hades anhalten not hallucination hallucination haunted homes is a british reality television series made by september films productions . [...] the show centers around writer richard hillier ( who owns the rights to the story ) , ghostwriter andrew scott smith ( pilot , only aired due to his lack of confidence level ) [...] they spend the weekend in a supposedly haunted house , hoping to find out if there are any ghosts around , [...] haunted homes ist eine britische realityfernsehserie , die von september films productions produziert wird . [...] im mittelpunkt der sendung stehen der autor richard hillier ( der die rechte an der geschichte besitzt ) , der ghostwriter andrew scott smith ( pilotfilm , der aufgrund seines mangelnden vertrauenslevels nur ausgestrahlt wurde ) [...] sie verbringen das wochenende in einem vermeintlichen spukhaus , in der hoffnung herauszufinden , ob es dort geister gibt , [...] ieva zunda ( born 20 july 1978 in tukums ) is a latvian athlete . [...] she did not make it past the first round at the 1999 and 2003 world championships . [...] in 2008 [...] shortly before the deadline -on 28 june , she had finally reached the qualifying standard in the 400 m ( 56 . 50 ) , as she clocked in the first round . she finished third in her heat , again missing out on a place in the first round . ieva zunda ( geboren am 20 . juli 1978 in tukums ) ist eine lettische leichtathletin . [...] bei den weltmeisterschaften 1999 und 2003 kam sie nicht über die erste runde hinaus . [...] 2008 versuchte sie erneut [...] kurz vor dem stichtag -am 28 . juni -hatte sie endlich die qualifikationsnorm über 400 m ( 56 . 50 ) erreicht , wie sie in der ersten runde lief . sie wurde dritte in ihrem lauf und verpasste erneut den einzug in die erste runde . word spans: [105, 105] word spans: [112, 114] word spans: [153, 153] word spans: [154, 154] ## results and discussions we present and discuss the downstream xlt results on anhalten for the token-level reference-free hallucination detection task across three xlt setups ( § 2.2): zero-shot transfer, few-shot transfer, and translate-train. zero-shot transfer. the results summarized in few-shot transfer and translate-train. as detailed in analysis. according to @xcite , nouns and verbs are the most frequently occurring partof-speech (pos) in the marked word spans of the hades dataset. the majority of instances with nouns (62.4%) and adjectives (74.0%) in the marked word spans belong to the hallucination class, while the majority of instances with verbs belong to the non-hallucination class (62.8%). this indicates a significant imbalance in label distribution. to assess the impact of this imbalance on cross-lingual transfer performance, we classify the validation set of anhalten based on the selected pos (nouns, verbs, adjectives) in the marked word spans. instances with marked word spans containing multiple words from different pos are excluded. to ensure an equal number of labels for each pos, we randomly remove instances from the more frequent class. this process results in 292 noun instances, 222 verb instances, and 62 adjective instances. we then analyze the xlt results of xlmr in the online setting. the pos results in we further conduct morphological analysis (detailed in appendix b) and demonstrate that preceding words indicate grammatical gender in german impact model performance, underscoring the importance of linguistic context. these findings emphasize the need to address imbalances and encourage future work to enhance model performance concerning diverse linguistic features for tokenlevel reference-free hallucination detection. ## conclusions token-level reference-free hallucination detection has predominantly focused on english, primarily due to the lack of robust benchmarks in other languages, hindering investigation into cross-lingual transfer approaches for this important task. to address this gap, we have presented anhalten, an extension of the english hades containing gold hallucination annotations in german, allowing for reliable and comparable cross-lingual estimates for token-level reference-free hallucination detection tasks. we utilized a modular adapter-based approach to facilitate the cross-lingual transfer, demonstrating the effectiveness of sample-efficient few-shot transfer. we believe that our dataset and findings advance the understanding of hallucination detection in cross-lingual transfer setups and contribute towards multilingual hallucination detection and real-time hallucination prevention in free-form text generation. search to include other types of hallucinations (e.g., subjective hallucinations) would provide a more comprehensive understanding of hallucination detection in various text types. we experimented on encoder-only multilingual plms, while decoderbased plms (e.g., le @xcite @xcite warrants exploration. we hope that future research builds on top of our findings and extends the research toward more domains, more languages, and specifically with the efficiency and effective concerns of hallucination detection in different languages.