Evaluating and Modeling Attribution for Cross-Lingual Question Answering (2023)

Abstract: Trustworthy answer content is abundant in many high-resource languages and is instantly accessible through question answering systems — yet this content can be hard to access for those that do not speak these languages. The leap forward in cross-lingual modeling quality offered by generative language models offers much promise, yet their raw generations often fall short in factuality. To improve trustworthiness in these systems, a promising direction is to attribute the answer to a retrieved source, possibly in a content-rich language different from the query. Our work is the first to study attribution for cross-lingual question answering. First, we collect data in 5 languages to assess the attribution level of a state-of-the-art cross-lingual QA system. To our surprise, we find that a substantial portion of the answers is not attributable to any retrieved passages (up to 50% of answers exactly matching a gold reference) despite the system being able to attend directly to the retrieved text. Second, to address this poor attribution level, we experiment with a wide range of attribution detection techniques. We find that Natural Language Inference models and PaLM 2 fine-tuned on a very small amount of attribution data can accurately detect attribution. With these models, we improve the attribution level of a cross-lingual QA system. Overall, we show that current academic generative cross-lingual QA systems have substantial shortcomings in attribution and we build tooling to mitigate these issues.

## introduction open-retrieval question answering (orqa) delivers promising performance for informationseeking question answering in about 20 languages @xcite @xcite . orqa models typically consist of a retriever that retrieves documents in a large corpus, followed by a generator that generates a short answer based on the top-ranked documents. recent work in orqa reached a new state of the art by not only retrieving documents in the same language as the query but by also retrieving passages cross-lingually, in additional languages @xcite . this approach is particularly beneficial for languages with limited online written content @xcite , potentially allowing users to access information that may not be available in their language. orqa models are typically evaluated with string-matching metrics (e.g., exact-match) based on extensive collections of question and answer pairs in multiple languages @xcite . however, these metrics are limited in three ways. first, they are inherently hard to scale to real-world applications, as they require the collection of gold answers for all the queries. second, in some cases, the answer can be correct without any overlap with a gold reference @xcite . third, short answers are usually not enough to provide a trustworthy answer, and users may prefer to access the underlying source document. to address this last challenge, @xcite framed a new task called attributed question answering (aqa). given a query, aqa consists of predicting a short answer along with a supporting document retrieved from a large corpus (e.g. wikipedia). this work is the first study on attributed question answering in the cross-lingual setting. 2 mea-figure 1 : attribution scenarios for cross-lingual open-retrieval question answering (xorqa). given a query in a source language, a retrieval model (mdpr) retrieves source language and cross-lingual documents, which are then used by a generation model, mgen, to produce a source language answer. for s1, in-language annotators assess attribution directly in the user's language while for s2, annotators validate attribution in english. we collect data for both scenarios in bengali, finnish, japanese, russian and telugu. based on this framework, we first measure the attribution level of cora @xcite , a stateof-the-art cross-lingual orqa system. we collect data in 5 languages: bengali, finnish, japanese, russian, and telugu. to our surprise, a large portion of generated answers was found not attributable to any retrieved passage. for instance, in japanese, up to 47% of answers exactly matching the gold reference are not attributable. this poor attribution may hurt the trust into our qa systems and limit their deployment. to improve the attribution level of cross-lingual qa systems, we experiment with a wide range of attribution detection models. we show that palm 2 in any language, possibly in a language different from the query, as a cross-lingual qa system. @xcite outperforms all the other models despite being fine-tuned on a very small sample of our collected data (250 examples). this result shows the potential of using large language models to create state-of-the-art cross-lingual attribution models using very little annotated data, allowing them to be inexpensively created for use in many of the languages of the world. additionally, we find that for bengali, finnish, japanese and russian, a t5 model fine-tuned on a large natural language inference corpora reaches very high accuracy compared to the baselines. our analysis shows that palm 2 can detect more than 86% of attributed answers that are not exactly matching the gold reference, showing that it is a useful alternative to exact match evaluation. these answers may be answers that are not captured by the gold references but that are alternative correct answers or they can be answers that are semantically equivalent to the gold reference but that do not overlap with the gold reference (e.g. different units). we discuss these cases in section 4.2. in summary, we make the following four contributions: (i) our work is the first to study attribution for question answering in a cross-lingual framework. we define two attribution scenarios, inlanguage attribution and in-english attribution and annotate approximately 10,000 examples in five languages; (ii) using this data, we evaluate the attribution of cora, a state-of-the-art cross-lingual qa system. we show that a large portion (7%-47%, depending on the language) of the answers are not attributable-neither to in-language passages nor to cross-language ones; (iii) we show that palm 2 and nli models can accurately detect attribution in the cross-lingual setting, significantly outperforming all other baselines and reaching above 90% accuracy for all 5 languages. moreover, our work is the first to approach it with large language models, and we show that with scarce amounts of data (250 examples), they can outperform nli models trained on millions of examples; and (iv) using our attribution detection model as a reranker, we show that we reach an average of +55% in attribution compared to a model with no reranking. ## the xor-attriqa dataset to the best of our knowledge, our work is the first to study the problem of attribution in the crosslingual setting. to make this study feasible, we collect the first multilingual attribution dataset. we use the attribution evaluation framework defined by @xcite . we hire bengali, finnish, japanese, russian, and telugu-speaking raters for the in-language scenario (s1) and english-speaking raters for the in-english scenario (s2). our analysis is based on the xor-tydiqa dataset @xcite in bengali, finnish, japanese, russian and telugu. to limit cost, we randomly sample about 50% of the validation set except for bengali and telugu (s1) annotations for which we take the entire set. we retrieve the passages and predict the answers using the cora system. we only evaluate the passages that are fed to the generator. for each (query, answer, passage) triplet, we ask three raters to answer "is the answer attributed to the passage?". 5 to ensure the quality of the data collected we report in in-english vs. in-language attribution as reported in in consequence, we will use the annotation from the in-language scenario (s1) as the gold labels to evaluate all our models (cf. 4 and 5). indeed, (s1) evaluates what a potential user may experience. so given the fact that (s1) is as good (or better) as (s2) concerning data quality, we decide to select the data from (s1). ## attribution evaluation of cora based on our newly collected xor-attriqa dataset, we now evaluate the attribution level of cora. ## attribution detection for xorqa so far, we found that state-of-the-art cross-lingual question answering systems lack attribution. we showed that a large portion of answers are not attributed to any passages, by collecting a large collection of attribution data in five languages. however, collecting attribution data is costly and time consuming. in a deployment setting, it would simply be infeasible to annotate every (query, answer, passage) triplet. in order to address this issue, we explore automatic attribution detection techniques. we build upon previous work on grounding and factual consistency in english @xcite . we also experiment with palm 2 @xcite a new state-of-the-art multilingual large language model (llm) in few-shot and scarce data (250 examples) settings. ## discussion and future directions language model-based nlp systems are making fast and continuous progress in generating fluent and helpful text. despite this progress, these models still make a lot of factual errors, specifically in languages different from english. attribution is the most promising approach in addressing this issue @xcite . our work finds that even the best xorqa system predictions lack attribution. these results can be explained by the tendency of these models to memorize facts @xcite and to hallucinate answers @xcite , which are in some cases correct. this shows that we need to make progress in detecting and selecting attributed sources that support the generated answer of cross-lingual qa systems. in this work, we proposed to use a large language model (palm 2) and natural language inference models to detect and 8 using translation ensures that everything is in the same language potentially improving the string-matching accuracy. our result points to two critical research directions to further make progress in informationseeking qa. first, we observed that in some languages (e.g., telugu), cross-lingual passages contribute very moderately to the attribution level. this shows that more progress is needed in cross-lingual retriever systems. second, we showed that stringmatching metrics based on gold references are inherently imperfect evaluation methods for qa and showed that palm 2 can be used to detect relevant attributed passages accurately with only small amounts of training data. this means that large language models-based attribution detection can potentially be used as evaluation metrics for qa in multiple languages. further work is needed to design robust llm-based metrics for cross-lingual information-seeking qa. ## conclusion by ensuring that the model predictions are supported by human-written text, attribution is one of the most promising ways to deploy nlp systems safely. in this work, we introduced and released the xor-attriqa dataset that includes approximately 10,000 examples in bengali, finnish, japanese, russian and telugu. thanks to xor-attriqa , we observe that state-of-the-art qa systems lack attribution in the cross-lingual setting. we showed that palm 2 and nli models are promising methods to detect attributed passages in 5 typologically diverse languages for information-seeking qa. having provided evidence for the lack of attribution in academic generative qa system, built tooling to detect and mitigate these issues, and releasing our collected attribution data in 5 languages, we hope to enable trustworthy cross-lingual qa systems to meet the information needs of people around the world.