CONCRETE : Improving Cross-lingual Fact-checking with Cross-lingual Retrieval (2022)

Abstract: Fact-checking has gained increasing attention due to the widespread of falsified information. Most fact-checking approaches focus on claims made in English only due to the data scarcity issue in other languages. The lack of fact-checking datasets in low-resource languages calls for an effective cross-lingual transfer technique for fact-checking. Additionally, trustworthy information in different languages can be complementary and helpful in verifying facts. To this end, we present the first fact-checking framework augmented with cross-lingual retrieval that aggregates evidence retrieved from multiple languages through a cross-lingual retriever. Given the absence of cross-lingual information retrieval datasets with claim-like queries, we train the retriever with our proposed Cross-lingual Inverse Cloze Task (X-ICT), a self-supervised algorithm that creates training instances by translating the title of a passage. The goal for X-ICT is to learn cross-lingual retrieval in which the model learns to identify the passage corresponding to a given translated title. On the X-Fact dataset, our approach achieves 2.23% absolute F1 improvement in the zero-shot cross-lingual setup over prior systems. The source code and data are publicly available at https://github.com/khuangaf/CONCRETE .

## introduction fact-checking is an important task that assesses the veracity of a claim. this task has gained increasing attention due to the widespread mis-and disinformation that has a significant socioeconomic impact on our society @xcite @xcite @xcite @xcite . while fact-checking is mainly conducted manually, especially in the journalism industry, with more than hundred of millions of social media posts and millions of blog posts published per day @xcite , manual fact-checking is no longer feasible. hence, we are urgently in need of reliable automated fact-checking approaches. most existing work develops fact-checking approaches on english-only corpus @xcite @xcite . one reason for this is the scarcity of fact-checking websites in other languages for constructing large enough non-english fact-checking datasets. therefore, it is even more challenging to build fact-checkers for low-resource languages. a solution is to leverage high-resource languages with zero-shot cross-lingual transfer, where the model is trained on source languages of richer resources and directly tested on target languages of lower resources. note that the sets of languages in the training set and the test sets are disjoint. with this technique, ground-truth labels for claims in low-resource languages are no longer required. few studies in the fact-checking literature have explored the cross-lingual setup. one line of work attempts to match an input claim with claims in other languages that have been verified @xcite . however, this approach fails when the claims have not been fact-checked in any language. another line of work builds classifiers with multilingual language models. for example, @xcite utilize google search to obtain snippets that are relevant to the input claim and train a model based on mbert @xcite in a cross-lingual setting. although google search excels at retrieving relevant information from a given claim, it disregards the trustworthiness of the information being retrieved. when a claim is erroneous, the search results often contradict each other, which likely leads to incorrect predictions, as shown later in section 5.3. motivated by these challenges, we propose con-crete, a claim-oriented coss-lingual retriever that retrieves evidence from a trustworthy multilingual passage collection for fact-checking. this approach can handle region-specific claims as long as relevant evidence is presented in the passage collection, which is much more accessible compared to similar claims in other languages. in addition, since it does not rely on a black-box retrieval system, our approach provides the flexibility to include only trustworthy information in the passage collection. one major challenge for training such a retriever is the lack of multilingual information retrieval (ir) dataset with claim-like queries. to this end, we propose a self-supervised crosslingual learning algorithm, cross-lingual inverse cloze task (x-ict), to learn the retriever based on pseudo-feedback. to mimic cross-lingual retrieval, we construct a pseudo-query for a given passage by translating its title into a randomly selected language. the objective for x-ict is to identify the passage corresponding to a given translated title among all candidate passages. since the title of news articles can often be regarded as a claim, this approach mitigates the domain discrepancy issue. as shown in figure 1 , our framework first performs cross-lingual retrieval to obtain evidence relevant to the input claim. then, a multilingual reader takes in the retrieved evidence and the input claim to classify the veracity of the claim. our contributions can be summarized as follows: ## task definitions the input is a claim c in an arbitrary language and the corresponding metadata, such as claimer and claim date. based on c, the retriever component of our model retrieves relevant passages p from a multilingual passage collection p where p can be in any language. then, the reader component takes in the claim c, the corresponding metadata, and relevant passages p to predict the veracity of c. note that the use of passage collection p to aid in fact-checking is a modeling choice, as no grounded evidence is available. we aim to build a passage collection that does not contain erroneous information. hence, we construct p by crawling 49,000 articles published in 7 languages foot_0 between september 2016 and december 2022 from a trustworthy news media, bbc.com, and split each article into passages. each passage contains at most 100 tokens, following @xcite . this results in a total of 347,557 passages. the collected passage collection has been made publicly available in the link mentioned in the abstract. ## proposed method our framework is a pipeline consisting of two components: (1) concrete, a claim-oriented crosslingual retriever that retrieves relevant passages from a multilingual passage collection, and (2) a multilingual reader that determines the veracity of a claim based on the compatibility of the claim and the passages retrieved. figure 1 shows an overview of the proposed method. the following sections describe each component in detail. ## conclusions and future work we have proposed concrete, a claim-oriented cross-lingual retriever that retrieves trustworthy passages from a multilingual passage collection. to overcome the lack of ir training data with claimlike queries, we present the cross-lingual inverse cloze task (x-ict) that leverages pseudo feedback to train the retriever. experimental results on x-fact showed that our approach outperforms all previous systems in the zero-shot cross-lingual setting. for future work, we plan to investigate the adaptive selection mechanism for passages based on distances and develop more robust readers for reasoning through passages of longer distances via representation learning.