Findings of the 2nd Shared Task on Multi-lingual Multi-task Information Retrieval at MRL 2024 (2024)

Abstract: Large language models (LLMs) demonstrate exceptional proficiency in both the comprehension and generation of textual data, particularly in English, a language for which extensive public benchmarks have been established across a wide range of natural language processing (NLP) tasks. Nonetheless, their performance in multilingual contexts and specialized domains remains less rigorously validated, raising questions about their reliability and generalizability across linguistically diverse and domain-specific settings. The second edition of the Shared Task on Multilingual Multitask Information Retrieval aims to provide a comprehensive and inclusive multilingual evaluation benchmark which aids assessing the ability of multilingual LLMs to capture logical, factual, or causal relationships within lengthy text contexts and generate language under sparse settings, particularly in scenarios with under-resourced languages. The shared task consists of two subtasks crucial to information retrieval: Named entity recognition (NER) and reading comprehension (RC), in 7 data-scarce languages: Azerbaijani, Swiss German, Turkish and , which previously lacked annotated resources in information retrieval tasks. This year specifally focus on the multiple-choice question answering evaluation setting which provides a more objective setting for comparing different methods across languages.

## introduction recent advancements in organizing online knowledge facilitated by large language models (llms) have fundamentally reshaped the way we approach information retrieval. this functionality creates exciting potential for new applications for education and media supporting seamless access to information on diverse subjects. however, this functionality is largely to limited in high-resourced languages, preventing equal access to potential applications in many under-resourced or studied languages across the world @xcite . recently, initiatives for creating standardized benchmarks for evaluating natural language processing (nlp) systems in a more linguistically inclusive setting had been proposed by corpora like xtreme @xcite and xtreme-up @xcite . although these data sets bring together large multilingual corpora they lack in generative human prepared data related to information access. the 2nd shared task on multi-lingual multi-task information retrieval (mmir), provides a benchmark for evaluating multi-lingual large language models (llms) in terms of their applicability for information retrieval in various under-resourced and typologically diverse languages. purely constructed using human annotated data consisting of examples of reading comprehension questions and named entity recognition in various context and languages, mmir benchmark presents a challenging new task for testing and improving llms. as evaluation resource we use wikipedia which we find representative of the inclusion of languages online. we pick five languages with varying degrees of resources and linguistic typology from three different language families: azerbaijani and turkish (turkic), igbo and yoruba, (niger-congo) and swiss german (germanic), and produce annotations in two tasks crucial for ir: named entity recognition (ner) and reading comprehension (rc). we present our data curation and annotation process as well as the findings of the evaluation in the resulting benchmark including prominent llms trained on multi-lingual multi-task settings: llama @xcite , aya @xcite and gemini @xcite . extending the data sets and competition from 2023, this year's edition allowed submissions both in open-ended and multiple-choice question answering to allow a more fine-grained and objective analysis. we received 3 submissions in the multiple-choice and 2 submissions in the open-ended rc tasks. the ner task also received 2 submissions. we provide more details on the data sets and a comparison of competing systems. narendrabhai damodardas modi ni mínśítà àgbà india ke . rìnlá àti mínísítà àgbà tí india ló . wó . ló . wó . lati o . dun 2014. o je . oloselu kan lati bharatiya janata party , agbari-is . e . oluyo . o . da ara ilu hindu kan. oun ni prime minister ako . ko . ni ita ti ileigbimo . jo . ti orile . -ede india lati s . e . gun awo . n ofin ite . lera meji pe . lu opoju to kun ati ekeji lati pari die . sii ju o . dun marun ni o . fiisi le . hin atal bihari vajpayee . per , loc , and org are in colours red, green, and blue respectively. we make use of label studio for annotation @xcite . ## languages ## data preparation the textual data for the generative task are based on wikimedia downloads foot_0 . rc annotations are prepared by sampling articles, splitting into paragraph-wise for question and answer annotations. in the extension of the benchmark this year, we annotate additional questions and wrong answer options for creating the multiple-choice qa setting @xcite . for the ne annotation, we ensure we sample only biographical articles and also only include articles available in all six languages. we use label studio for rc and ner annotation @xcite with the tag set (person (per), organization (org), location (loc)) and ensure an annotation overlap of 2% for ner. the question-answer pairs were always produced from two separate annotators. we recruited two annotators per language, for ig and tr respectively four annotators contributed, and five persons annotated yo. the resulting data statistics for the validation and test splits can be found in ## results we evaluate the overall system performance on the generative task using automatic metrics weighted by the number of articles in the test set containing individual context used for answering the rc questions ## conclusion and future work we presented a new multi-lingual multi-task benchmark on information retrieval from wikipedia in five languages from typologically-diverse and lowresourced language families in the open-ended or multiple-choice qa and ner tasks. we organized a shared task to call for system development on this challenging benchmark where we conducted a detailed analysis on how state-of-the-art llms perform in language understanding and generation under low-resourced settings. in addition to finding strong evidence on fall backs in both understanding and generation capabilities of llms in low-resourced languages, we also find it crucial to invest in better automatic evaluation metrics for generation in different languages. while we do not find this task to be solved, we plan to keep the competition open and promote more investment into the progress of information retrieval for languages with non-prominent and low-resourced characteristics.