M c G ill NLP Group Submission to the MRL 2024 Shared Task: Ensembling Enhances Effectiveness of Multilingual Small LM s (2024)

Abstract: We present our systems for the three tasks and five languages included in the MRL 2024 Shared Task on Multilingual Multi-task Information Retrieval: (1) Named Entity Recognition, (2) Free-form Question Answering, and (3) Multiple-choice Question Answering. For each task, we explored the impact of selecting different multilingual language models for fine-tuning across various target languages, and implemented an ensemble system that generates final outputs based on predictions from multiple fine-tuned models. All models are large language models fine-tuned on task-specific data. Our experimental results show that a more balanced dataset would yield better results. However, when training data for certain languages are scarce, fine-tuning on a large amount of English data supplemented by a small amount of “triggering data” in the target language can produce decent results.

## introduction in this paper, we present our submission for the mrl 2024 shared task 2 . the shared task includes the following three tasks: named entity recognition (ner), free-form question answering (ffqa), and multiple-choice question answering (mcqa). each task involves a final test set for five languages: igbo, swiss german, turkish, azerbaijani, and yorùbá. our systems are designed to support all of these languages simultaneously. our systems leveraged the remarkable success of transformer-based @xcite , pretrained language models (lms) such as bert @xcite and t5 @xcite , which have demonstrated outstanding performance in various natural language processing (nlp) tasks in recent years. these models, with their large number of parameters and pre-training on vast datasets, have proven to be highly effective in extracting and representing information possessed by input sequences @xcite . their strong generalization capabilities make them well-suited for fine-tuning on specific tasks, such as ner and translation. multilingual pre-trained llms, like xlm-roberta @xcite , mt5 @xcite , and their variants, which were trained on extensive multilingual datasets, are particularly effective for multilingual tasks. these models capture semantic structures/knowledge shared across languages, enhancing their ability to transfer knowledge between languages. fine-tuning these models for specific tasks allowed us to fully utilize their rich token-level and sentence-level semantic representations, which are essential for tasks requiring detailed language understanding. for instance, ner benefits from the token-level granularity learned during pretraining @xcite , while ffqa and mcqa require robust sentencelevel comprehension, which these models provide @xcite . the combination of pre-training on extensive multilingual corpora and task-specific fine-tuning enabled our system to achieve decent performance across all five target languages. during the fine-tuning phase, in addition to hyper-parameter selection, our systems employed other strategies to promote a smoother and fasterconverging learning process, such as using data from languages closely related to the target languages, applying curriculum learning @xcite , and interleaving data from various languages to enhance model performance and smooth the learning process. the experiment results show that different base models with a similar number of parameters exhibit varying advantages for different languages after fine-tuning. ensembling the outputs from each model results in better and more robust overall performance. additionally, given the limited availability of data for certain languages, leveraging large amounts of task-specific data in english-which is the easiest to obtain-along with smaller amounts of data in the target language, allows knowledge transfer from english to the target language. this approach outperforms fine-tuning exclusively on the limited data available in the target language. ## background in this section, we provide a brief overview of the background knowledge for the three tasks involved in the shared task, along with the three techniques we employed to facilitate the learning process. named entity recognition ner is an nlp task that focuses on identifying and categorizing specific tokens or phrases in a text as belonging to predefined entity types, such as persons (per), organizations (org), locations (loc), dates (date), and other relevant categories. a named entity refers to a real-world object or concept that can be recognized by its proper name within the text. for example, in the sentence "barack obama visited paris in 2015," the named entities are "barack obama" (person), "paris" (location), and "2015" (date). in this shared task, we only consider three entity tags: persons, organizations, and locations. free-form question answering ffqa involves providing answers to natural language questions based on the information given. this task assumes an information-seeking scenario, where users ask questions without knowing the answer in advance, and the system is responsible for finding a relevant answer based on information presented in the passage (if one exists). in this task, the system is given a question, a title, and a passage, and must either generate a text sequence for the correct answer or indicate that there is no answer for the question based on information available in the passage by generating the text sequence "no answer". for example, consider the passage: "tom went to the supermarket and bought two apples." if the question is "what did tom buy in the supermarket?", the system should return the answer "two apples." however, if the question is "which supermarket did tom visit?", the system should respond with "no answer," as the passage does not specify the name of the supermarket. multiple-choice question answering similar to ffqa, mcqa assumes a scenario where users seek information by asking questions without knowing the answer and are given a question, title, and passage. however, unlike ffqa, the mcqa system is also provided with four potential options, and its task is to identify the correct one based on the information in the passage. for instance, consider the passage: "tom went to the supermarket and bought two apples." if the question is "how many apples did tom buy?" and the four options are "a. 1", "b. 2", "c. 3", and "d. 4", the system should return "b". curriculum learning curriculum learning @xcite is a machine learning strategy that gradually introduces a model to progressively more challenging data pieces over multiple training iterations. this method can often produce better results compared to using a randomly shuffled training set. this approach is effective in the sense that, the model begins by learning general concepts through simpler examples, and then incrementally incorporates more detailed and complex information as more difficult examples are introduced. for our systems, we define "difficulty" by the length of the input text, where longer text equates to greater complexity and comes later in the epoch, as shown in figure 1 . since curriculum learning is a paradigm that focuses solely on the selection and ordering of training data, it can be integrated with various other machine learning techniques, like interleaving multilingual data pieces which we will introduce later in this section. knowledge transfer knowledge transfer in multilingual llms refers to the model's ability to leverage information, patterns, or representations learned in one language to enhance its performance or understanding in another. this happens because multilingual llms develop shared representations of concepts that can be applied across different languages. to facilitate the knowledge transfer for our base models, we fine-tuned the base models on diverse multilingual data. this includes a relatively small amount of data for the target languages, additional data for languages closely related to the target languages, and a large amount of data from high-resource languages like english. interleaving multilingual data pieces interleaving multilingual data pieces is a machine learning technique used to train multilingual models by interleaving data from various languages during training. this approach promotes cross-lingual knowledge transfer by encouraging the model to develop shared linguistic representations and structures, which improves its ability to generalize across languages. it is especially effective in crosslingual information retrieval scenarios, allowing the model to utilize common features across languages and enhance performance in low-resource language settings. an illustration of this approach can be found in figure 2 3 methods in this section, we provide a detailed illustration of each system we implemented for the three tasks involved in mrl 2024. ## experiment in this section, we provide detailed information about our implementation, including the computational resources used to run the experiments, the specifics of the training process, and the datasets used to train the models for each of the three tasks. additionally, we will present the results on the test set provided by the organizers of this shared task, along with an analysis of the experimental results. ## conclusion our study on multilingual multi-task information retrieval revealed key insights across ner, ffqa, and mcqa tasks. in the mcqa task, our ensemble models demonstrated particular strength, outperforming individual fine-tuned models. this underscores the benefits of combining predictions from multiple models to boost accuracy and robustness. for ner, our system showed strong performance across most languages, achieving the highest f1 scores in four out of five languages compared to the other participating systems. however, we observed a significant performance drop for igbo, highlighting the challenges of consistent performance across diverse languages. we observed variable performance across tasks, with challenges particularly evident in ffqa and significant differences across languages, especially in low-resource settings. this variability was also present in ner, where our model's performance on igbo lagged significantly behind other languages. looking forward, our findings suggest several promising areas for improvement. enhancing ffqa performance through better fine-tuning strategies and exploring cross-lingual transfer methods is crucial. developing task-specific model architectures that can better capture the nuances of each task while maintaining multilingual capabilities could lead to significant advances. improv-ing data augmentation and efficient fine-tuning approaches, especially for low-resource languages, remains a key challenge. increasing model interpretability will be vital to better understand and address performance discrepancies across languages and tasks. for ner, investigating the causes of the performance anomaly in igbo and refining our ensemble method could create a more universally effective model across diverse languages. while our approach shows promise, particularly for mcqa and most languages in ner, there is substantial room for further research. the goal remains to develop robust, multilingual, multi-task information retrieval systems that can overcome language barriers, address performance inconsistencies, and improve access to global information across a wide range of languages and task types.