MELAB enchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource M altese NLP (2025)

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across various Natural Language Processing (NLP) tasks, largely due to their generalisability and ability to perform tasks without additional training. However, their effectiveness for low-resource languages remains limited. In this study, we evaluate the performance of 55 publicly available LLMs on Maltese, a low-resource language, using a newly introduced benchmark covering 11 discriminative and generative tasks. Our experiments highlight that many models perform poorly, particularly on generative tasks, and that smaller fine-tuned models often perform better across all tasks. From our multidimensional analysis, we investigate various factors impacting performance. We conclude that prior exposure to Maltese during pre-training and instruction-tuning emerges as the most important factor. We also examine the trade-offs between fine-tuning and prompting, highlighting that while fine-tuning requires a higher initial cost, it yields better performance and lower inference costs. Through this work, we aim to highlight the need for more inclusive language technologies and recommend for researchers working with low-resource languages to consider more “traditional” language modelling approaches.

## introduction large language models (llms) have seen a huge rise in use due to their strong performance and remarkable generalisability across a wide range of diverse tasks @xcite @xcite . their appeal is evident from their ability to perform tasks without additional training, with models able to infer from a few examples (few-shot or in-context learning) or an instruction (zero-shot) @xcite . furthermore, the availability of a plethora of multilingual models makes this technology more accessible for many languages, due to the inherent cross-lingual transfer capabilities. despite this, many low-resource languages still face significant challenges in achieving strong performance with these models @xcite . prior research on older multilingual models, such as mbert, highlighted the challenges faced by low-resource languages, particularly when a language is absent from pre-training @xcite . however, modern llms often have an additional training phase, designed to improve their generalisability: instruction-tuning. this raises new questions about the extent to which instruction-tuning mitigates or exacerbates performance gaps for low-resource languages. in this work, we aim to address this for maltese, an official eu language that ranks the lowest in the digital language equality score @xcite . while our primary objective is to understand which llm properties influence downstream task performance, we compare this to more traditional fine-tuning approach with relatively smaller models. the main contributions of this work are: ## overall trends the starting point of our analysis is to understand the performance of all individual models across tasks by performing zero-shot prompting. thus, we aggregate the scores of each model averaged across discriminative and generative tasks. 4 for each model, we then calculate an overall score across all tasks by averaging these two scores. the results are shown in figure 1 . overall, aya-101 is the best-performing model, followed closely by mt0-xxl. we attribute this primarily to the models' exposure to maltese data, which we further investigate in section 4.1. the smaller mt0 models are the next best-performing models overall, along with salamandra instruct 7b, gemma 2 instruct 9b, and llama 3 instruct 8b. for generative tasks, aya-101 performs better than any other model, often by a significant margin. this is followed by mt0-xxl, eurollm instruct 1.7b, teuken research instruct 7b, salamandra instruct 7b, and llama 3 instruct 8b. in the case of discriminative tasks, on average, mt0-xxl and gemma 2 instruct 9b perform better than aya-101. we take a closer look at the score distribution for each task in figure 2 . more models are competitive with the fine-tuned baselines on discriminative tasks than generative tasks. performance on generative tasks hovers near 0 for many models. a more in-depth analysis of the individual models on each task (see figure 9 ) reveals that for generative tasks, the top performers are aya-101, mt0-xxl, and teuken instruct research, and to a lesser degree, eurollm 1.7b and llama 3 instruct 8b. these models often act as outliers from the rest of the models. the insights highlight that generating text data is much more challenging than understanding it and that many models fail to capture the linguistic nuances of a low-resource language like maltese. looking at each task in figure 2 , we observe that models generally struggle with taxi1500 and mul-tieurlex among discriminative tasks and eur-lex-sum among generative tasks. this could be attributed to the specific domain of these tasks: the bible for taxi1500 and european union documents for multieurlex and eur-lex-sum. the latter two tasks have input sequences that are also generally longer, which hampers performance due to the model's limited context length. moreover, we note that multieurlex is the only discriminative task where models perform quite on par with one another, particularly since it is a multi-label classification task on a massive scale. when compared to the baselines, all prompted models perform worse, with the exception of the sentiment analysis task, for which some models outperform mbert. among the baselines, mt5 generally performs better than mbert, except for multieurlex, potentially due to the task being illsuited for generative models, as discussed earlier. overall, bertu performs the best in discriminative tasks due to its maltese pre-training, except for sentiment analysis for which we observe a perfect score for mt5. while we are uncertain why the model does so well, we posit that this is due to the task being quite simplistic, since it is the only task where prompted models outperform some of the baselines. moreover, out of all the tasks, this dataset is the most likely to contain code-switching, which might make it easier for multilingual models to pick up on certain linguistic signals. ## effect of maltese exposure we now examine the effect that exposing the model to maltese has on its performance. apart from a model's overall training (pt or it), models are further grouped into different categories based on their explicit training on maltese data, indicated by: no, pt, and it, referring to no exposure to maltese data, exposure during pre-training, and exposure during instruction-tuning, respectively. for these analyses, we exclude models for which this information cannot be inferred from publicly available metadata (nk). ## efficiency the computational efficiency of fine-tuning and prompting is also an important factor to consider. we select the following prompted models based on the best overall performance and the model architecture variety: aya-101, mt0-xxl, and llama 3 instruct 8b. we consider all fine-tuned models: bertu, mbert, and mt5-small. to estimate the computational requirements, we follow @xcite and compute the floating-point operations per second (flops, @xcite required for a single instance. we take the median sequence length of an instance for each dataset and use it to calculate the inference flops for a given model. for fine-tuned models, we use only the raw input sequence in textual format. for prompted models, we also include the accompanying instruction. for the purpose of this analysis, as expected, the inference cost of fine-tuned models is magnitudes smaller than that of prompted models due to the smaller model sizes. we also note that for prompted models, as the number of few-shot examples increases, so does the inference cost. the cost for fine-tuning models is also magnitudes larger than applying inference on a single instance. however, as we apply inference on more examples, this initial upfront cost diminishes. if we define efficiency as a function of the overall performance, the cost, and the number of inference samples as follows: due to the lack of research on generative maltese nlp, our research is primarily related to various multilingual benchmarking and evaluation works, although none of them include maltese. ahuja et al. (2023) evaluate llm performance on a newly developed benchmark covering 70 languages. they 0 200 400 600 800 1000 number of samples 10 -15 10 -14 10 -13 10 -12 10 -11 efficiency bertu 0.126 mbert 0.179 mt5 0.3 mt0 13.0 aya-101 12.9 llama 3 instruct 8.0 find that low-resource languages are negatively affected by ill-fitted tokenisers and limited pretraining data used by the model. @xcite study the impact of few-shot prompting across 56 languages, finding that the few-shot performance does not always improve over zero-shot, depending on the model, task, and language. @xcite similarly analyse few-shot prompting performance but in a cross-lingual setup, finding that while fewshot generally helps, models perform particularly poorly on lower-resourced languages. @xcite analyse the performance on code-switched languages. in their analysis, they find that both an increased number of shots and model size (when considering the same model family) generally improve performance. most of these works also fine-tune smaller bert-based and/or mt5 models, similarly finding that they outperform prompted llms @xcite @xcite . @xcite find that chatgpt underperforms supervised sota models on most tasks. our research is complementary to these works as we look closer at the performance of a single language. we also significantly scale the number of language models which allows us to study multiple factors affecting model performance. ## conclusion in this paper, we present a comprehensive evaluation of llms for maltese, a low-resource language. we introduce a new benchmark covering a total of 11 discriminative and generative tasks. we carried out an evaluation of 55 publicly available models under zero-shot and one-shot prompting, revealing significant performance variations. our study highlights the need to improve the state of low-resource languages such as maltese. in analysing our results, we uncover several key trends. prompted models consistently lag behind smaller fine-tuned models, particularly in generative tasks where their performance is significantly lower. crucially, the level of exposure to maltese in a model's training has the largest bearing on performance, especially in instruction-tuning. additionally, our efficiency analysis highlights the trade-offs between fine-tuning and prompting. while fine-tuning incurs higher initial costs, the inference cost is significantly lower. the initial cost incurred to fine-tune smaller models quickly pays off with more inference instances, apart from the higher performance on downstream tasks. these findings underscore the pressing need for more inclusive language models that better support low-resource languages at every stage of the training pipeline. while llms offer strong generalisability, their limited performance on low-resource languages like maltese reduces their usability in these scenarios. for researchers with limited computational resources, fine-tuning smaller models presents a viable alternative to prompting larger models, despite the trade-off in generalisability. ultimately, our study calls for a more balanced approach to model development, ensuring that lowresource languages like maltese receive adequate representation in the evolving landscape of llms.