Multilingual Fine-Grained News Headline Hallucination Detection (2024)

Abstract: The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the “hallucination” problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand <article, headline> pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset’s challenges and utilities. Second, we test various large language models’ in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.

## introduction a news headline provides a concise summary of its corresponding news article, enabling readers to quickly grasp the essence of a news story. numerous generative models @xcite @xcite have been developed to automate the process of condensing a news article into its headline, achieving generally commendable quality. however, people note that these models often encounter the hallucination problem, where the produced headline does not fully align with the source article's content. for example, as shown in figure 1 , the generation model is given an article about "ua adds new routes in midwest" and outputs the headline "ua displays more interests in routes from midwest, europe". this headline is considered as a hallucination because it references "europe" which is absent from the article, and omits the crucial detail that "new routes in the midwest are being added". to mitigate these hallucinations, various studies propose to pre-process the training corpus of generation models by removing or re-weighting possibly hallucinated examples @xcite @xcite . another line of work proposes to first detect hallucinations in generated outputs and then filter them in a postprocessing stage @xcite . these approaches typically adopt a binary or three-way classification scheme and focus on examples written exclusively in english or translated from a single english source. despite some promising results, it remains unclear how these approaches can capture more fine-grained hallucination error types in multilingual news articles and inform more nuanced decision making process. in this work, we propose a new task -finegrained headline hallucination detection and study it in the multilingual setting. this objective is to identify a set of fine-grained entailment relations between a given news article and its headline. taking the example in figure 1 for instance, we aim to advance beyond a simple "neutral" classification and provide more fine-grained predictions: (1) "unsupported information" due to the article's lack of references to "routes from europe", and (2) "missing key information" because the headline fails to capture a core message of the article -the introduction of new routes in the midwest. to advance research in this area, we introduce the first multilingual fine-grained headline hallucination detection (mfhhd) dataset, featuring 11,469 examples across 5 languages. each example comprises a news article, a generated news headline, a coarse-grained hallucination label, and a set of fine-grained hallucination labels annotated by 2 to 4 dedicated domain experts fluent in the language of the original article. additionally, for examples labeled as "neutral" or "contradict", annotators will provide a natural language justification for their fine-grained annotations. the introduction of this new mfhhd dataset presents intriguing research challenges, such as identifying complex, nuanced types of hallucination errors and exploring whether existing hallucination detection moethods, previously englishcentric, can be adapted for multilingual use. to answer these questions, we carry out extensive experiments in both supervised fine-tuning and fewshot learning scenarios. in the supervised finetuning context, we find that model pre-training on natural language inference datasets and incorporation of natural language explanations into seq2seq based classifier can both enhance the detection performance. in the few-shot learning domain, we evaluate various large language models (e.g., chatgpt (openai, 2022), palm2 variants @xcite ) and observe that they perform worse than the smaller fine-tuned models (e.g., mt5-xxl @xcite ). to improve these llms' in-context learning capabilities, we introduce two prompting techniques: (1) language-dependent demonstration selection which dynamically chooses few-shot examples in the same language as the test query example, and (2) coarseto-fine prompting that guides llms to generate a coarse-grained prediction before making finegrained hallucination type predictions. both techniques can significantly enhance llm's few-shot learning effectiveness and boost the detection performance in terms of the example-f1 metric. contributions. the major contributions of this paper are summarized as follows: (1) we introduce a novel task, fine-grained headline hallucination detection, aimed at identifying more nuanced hallucination error types; (2) we create a new multilingual fine-grained hallucination detection dataset mfhhd, curated by news domain experts; and (3) we conduct extensive experiments on the mfhhd dataset, delving into its complexities and offering valuable insights for improving the accuracy of finegrained hallucination detection in both supervised and few-shot learning settings. ## problem formulation in this study, we represent both a news article d and a news headline h as a sequence of tokens. the ideal purpose of a news headline is to provide a concise summary of the news article. however, when automatically generated, the headline can hallucinate and misrepresent the original intent of the source article. the coarse-grained headline hallucination detection task inputs a pair of news article and headline d i , h i and outputs its coarsegrained entailment relation l i indicating whether the headline is fully supported, directly contradicts, or remains neutral with respect to the article. in many real-world applications, we notice this the three-way entailment classification schema is too coarse-grained and fails to pinpoint the exact hallucination reasons (e.g., the headline includes an incorrect number or reports a person's subjective opinion as a fact). therefore, we propose the fine-grained headline hallucination detection task that returns a set of fine-grained entailment relations r @xmath0, r 2 i , . . . } for a pair of input article and headline d i , h i . each relation r j i specifies a detailed reason why the given headline h i either supports, contradicts, or is neutral in relation to the corresponding article d i . ## mfhhd dataset in this work, we collect the first multilingual fine-grained headline hallucination detection (mfhhd) dataset that contains 11,469 examples across 5 languages (english, spanish, german, french, portuguese). each example includes a news article, a news headline, a coarse-grained hallucination label, a set of fine-grained hallucination labels, and an optional set of natural language annotation justifications. the dataset is currently available at: https://bit.ly/ mfhhd-dataset . ## supervised fine-grained headline hallucination detection in this section, we experiment a set of supervised methods for multilingual fine-grained headline hallucination detection. the goal is to better understand the characteristics and challenges of our mfhhd dataset and to share some valuable insights that can help later llm-based few-shot method designs (c.f. section 5). ## few-shot fine-grained headline hallucination detection we introduce various few-shot methods that leverages large language models for detecting finegrained headline hallucinations in this section. ## conclusions and future work this study explores multilingual fine-grained headline hallucination detection and introduces the mfhhd dataset -a collection of over 11,000 expert-annotated examples across 5 languages with natural language explanations. through extensive experiments, we discover that supervised models gain from pre-training on nli datasets and the integration of explanations into their outputs. additionally, llm few-shot learners show improved performance when utilizing language-dependent demonstration selection and adopting a coarseto-fine prompting strategy. interesting future research directions include (1) employing parameterefficient tuning techniques to directly train llms on the mfhhd dataset, (2) annotating some finegrained hallucination classes at the span level, and