CUNI and LMU Submission to the MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval (2024)

Abstract: We present the joint CUNI and LMU submission to the MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval.The shared task objective was to explore how we can deploy modern methods in NLP in multi-lingual low-resource settings, tested on two sub-tasks: Named-entity recognition and question answering.Our solutions to the subtasks are based on data acquisition and model adaptation.We compare the performance of our submitted systems with the translate-test approachwhich proved to be the most useful in the previous edition of the shared task.Our results show that using more data as well as fine-tuning recent multilingual pre-trained models leads to considerable improvements over the translate-test baseline.Our code is available at https://github.com/ufal/mrl2024-multilingual-ir-shared-task.

## introduction over the past few years, large language models (llms) have attracted a fair share of attention from the research community. this is mainly caused by the remarkable in-context learning properties these models exhibit, especially in languages where there is plenty of data available @xcite . very recently, research advances have shown promising results in low-resource language processing by leveraging llms trained primarily on english data @xcite . meanwhile, massively multilingual approaches also started to deliver good results @xcite . the mrl 2024 shared task on multi-lingual multitask information retrieval aims to build upon this trend in tasks of named-entity recognition (ner; § 2) and question answering (qa; § 3) for alsatian, azerbaijani, igbo, indonesian, turkish, uzbek, and yoruba. in both subtasks, our submissions include finetuned mulitlingual models, compared to a translatetest baseline @xcite . ## named entity recognition the goal of the ner subtask was to detect and classify words and phrases into one of three categories: person (per), organization (org), and location (loc). unlike the previous year's edition, date and time entities were omitted from the task. for development, the organizers released validation data in alsatian, azerbaijani, turkish, and yoruba, each of around 120 sentences. we experiment with the translate-test approach and compare it with the most recent massively multilingual models ( § 2.1). we collect additional training data for the shared languages ( § 2.2) and finetune the best-scoring multilingual model ( § 2.3). ## question answering the goal of this task is to answer questions within a given context in two scenarios: first, select the correct answer from a set of four choices (multiplechoice questions). second, generate a free-form answer in natural language (open questions). the organizers provided 200 multiple-choice questions for all languages except uzbek. all correct options in the development data were labeled as "a". to balance the dataset, we shuffle the ordering of the options in the data and report the performance on this shuffled dataset. additionally, around 100 single-reference open questions for all languages were provided. we experiment with llms in the zero-shot setup both in the task languages and when translating the test into english ( § 3.1). then, we collect qa datasets that are available for the task languages ( § 3.2). we use the data to fine-tune the models ( § 3.3). finally, we experiment with ensembling of the models outputs in the zero-shot setup ( § 3.4). ## conclusions we presented our submissions to the mrl shared task on multi-lingual multi-task information retrieval. our methods based on data acquisition and fine-tuning of multilingual pre-trained models achieve good results compared to the translate-test approach, which was the key idea of the winning system from 2023 @xcite . for ner, we achieved our best results by finetuning state-of-the-art models specifically for the shared task languages and entities. in the qa subtask, we achieved our best results using the llms in the zero-shot setup.