An Analysis of Multilingual FA ct S core (2024)

Abstract: FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work in studying the behavior of FActScore in other languages. This paper studies the limitations of each component in the four-component pipeline of FActScore in the multilingual setting. We introduce a new dataset for FActScore on texts generated by strong multilingual LLMs. Our evaluation shows that LLMs exhibit distinct behaviors in both fact extraction and fact scoring tasks. No LLM produces consistent and reliable FActScore across languages of varying levels of resources. We also find that the knowledge source plays an important role in the quality of the estimated FActScore. Using Wikipedia as the knowledge source may hinder the true FActScore of long-form text due to its limited coverage in medium- and low-resource languages. We also incorporate 3 mitigations to our knowledge source that ultimately improve FActScore estimation across all languages.

## introduction recent advancements in llms have demonstrated significant capabilities @xcite @xcite @xcite in many applications @xcite . despite this advancement, llms remain prone to generate false information in response to information-seeking queries @xcite . to address this critical problem, llms have been trained at unprecedented scales @xcite to cope with the massive world knowledge and aligned to reduce hallucination @xcite @xcite . to further prevent the generation of false information, the retrieval augmented generation method provides retrieved documents from trustworthy sources to the llm @xcite . factscore was introduced to estimate the factuality of generated texts automatically @xcite and at a low cost by combining llm-asa-judge scoring @xcite with existing reliable knowledge sources such as wikipedia. factscore has been enhanced to incorporate a larger knowledge base, like the internet, and to utilize more powerful retrieval models such as google search, resulting in better estimation across a larger domain coverage @xcite . with the rapid development of multilingual llms @xcite , many more people are interacting with llms in an increasingly diverse set of languages. hence, there is a crucial need to monitor and improve the factuality of texts beyond just the english language, making it helpful and safe for users across the entire world @xcite . in this paper, we study the feasibility of the factscore pipeline @xcite in a multilingual setting. the factscore pipeline consists of multiple components: a knowledge source, a retrieval model, an llm-based fact extractor, and an llm-based fact scorer. we aim to scrutinize each component individually to identify bottlenecks and address these issues. however, there is no existing multilingual dataset for evaluating factscore besides the original english-only dataset published by @xcite . to bridge this gap, we annotate a new native dataset of factuality in 3 non-english languages representing high-, medium-, and lowresource levels. this dataset is created on the texts generated by strong multilingual llms, i.e., gpt-4 and gemini-pro-1.0. we find that all evaluator models show decreased factscore accuracy in lower-resource languages. we attribute this to several components. first, the performance of fact extraction, the simplest task in the factscore pipeline, deteriorates with lower resource languages. to address this issue, we finetuned an open-source llm for this task and achieved better performance than gpt-3.5. second, the quality of the knowledge source is crucial to the overall accuracy of factscore. higher resource languages typically have wikipedia pages with higher quality and coverage, leading to better factscore estimation. using the internet as the knowledge source @xcite , therefore, has the greatest impact on improving the accuracy of factscore estimation in medium and low-resource languages. our contributions are as follows: ## related work with the advancement of language model development, numerous methods have been proposed to assess their factual alignment. a significant portion of these efforts involves using questions and corresponding short answers @xcite , slot-filling @xcite task related to specific pre-collected factoids, however, they do not reflect practical use cases @xcite . instead, directly assessing open-ended generated texts offers a clearer signal of the level of factuality in real use cases @xcite . @xcite estimate the factscore of biographies generated by llms by evaluating individual candidate facts in the text. @xcite extended topic coverage and utilized the google api to query references for evaluation, thereby accessing a broader range of domains. our study builds heavily on these approaches, focusing on the effectiveness of factscore across high-, medium-, and low-resource languages. in these scenarios, both the language models' performance in each component of the evaluation pipeline and their multilingual capabilities are critical. other approaches rely on language models' internal knowledge pools for factuality assessment @xcite . while this approach offers simplicity, it raises concerns about the intrinsic factual alignment of these evaluators. considering multilingual factuality, x-factr @xcite and mlama @xcite , adapted from lama @xcite , assess models' relational knowledge through the "fill-in-the-blank" task. x-fact (gupta and srikumar, 2021) releases a multilingual fact-checking benchmark, a factual correctness classification task covering various topics and 25 typologically diverse languages across 11 language families. @xcite introduces an extension of mlama and x-factr and a new metric to assess the crosslingual consistency of language models. while these attempts shed light on multilingual factuality alignment, they mainly involve pre-collected sets of factual statements. our work aims to evaluate the factuality of open-ended text generation. @xcite adapted factscore for a multilingual context by translating the biographies to english. our work investigates both translation and performing the entire factscore pipeline directly in the reference language. we also designed a comprehensive set of biographies to better capture the cultural proclivities of the target population. ## tasks & resources in this work, we evaluate the factscore in multilingual settings using two resources: a translated annotation from previous work and a new native annotation. ## mitigations the previous sections have shown evidence of a correlation between lower resource languages, lower retrieval performance (see we use the 32 generated biographies in the three studied languages that we used to assess knowledge sources in section 4.3. we consider the facts annotated by native speakers using the whole internet as the golden data. we evaluate these techniques by measuring their scoring accuracy with the golden labels. ## conclusion this paper scrutinizes the factscore pipeline for long-form generated texts in the multilingual setting. we generated new fact candidates and annotated a new corpus for factscore evaluation. the most recent open-source llms struggle with the atomic fact extraction task. finetuning on this task can match the performance of much larger closesource models, e.g., gpt3.5. more importantly, the fact scoring task is very sensitive to the coverage of the knowledge source. although wikipedia is reliable, it lacks coverage in lower-resource languages, which leads to a severe underestimation of the factscore. we show that mitigation such as extending the knowledge source through increasing the amount of wikipedia data, allowing access to the internet, and even augmenting low-coverage wikipedia articles with unverified text generated by an llm improve multilingual factscore esti-mation.