Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks (2025)

Abstract: This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Building upon the ScandEval benchmark, initially restricted to evaluating encoder models, we extend the evaluation framework to include decoder models. We introduce a method for evaluating decoder models on NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic, Faroese, German, Dutch, and English. Through a series of experiments and analyses, we also address research questions regarding the comparative performance of encoder and decoder models, the impact of NLU task types, and the variation across language resources. Our findings reveal that encoder models can achieve significantly better NLU performance than decoder models despite having orders of magnitude fewer parameters. Additionally, we investigate the correlation between decoders and task performance via a UMAP analysis, shedding light on the unique capabilities of decoder and encoder models. This study contributes to a deeper understanding of language model paradigms in NLU tasks and provides valuable insights for model selection and evaluation in multilingual settings.

## introduction language models have attained remarkable natural language understanding (nlu) performance, both with encoder-based architectures like bert @xcite and and decoder-based architectures like gpt-3 @xcite . the encoder models have excelled in capturing contextual information for downstream tasks through masked language modeling objectives, while decoder models have shown strong generative capabilities by autoregressively predicting subsequent tokens based on preceding context. since the "chatgpt boom" in 2023, the research community has been increasingly focused on decoder models @xcite for both natural language generation (nlg) and nlu tasks. however, few studies have systematically compared the performance of encoder and decoder models across a diverse range of nlu tasks, and the studies that exist have primarily focused on english. this leaves a gap in our understanding of how the two language model paradigms perform in multilingual settings across different languages and tasks. nielsen (2023) introduced the scandeval benchmark and evaluated encoder language models on four different natural language understanding tasks in danish, swedish, norwegian (bokmål and nynorsk), icelandic and faroese. in this paper, we bridge this gap by extending the scandeval benchmark to encompass the evaluation of decoder models on multilingual nlu tasks, as well as expanding the language resources to include german, dutch and english. our main research question is which language model paradigm is better suited for nlu? we will answer this question with the languages danish, swedish, norwegian, icelandic, faroese, german, dutch and english as a case study. to concretise our main question, we will study the following research questions in this paper: our main contributions of this paper are the following: ## datasets in this section we present the datasets that we are evaluating the models on, all of which are now included in the scandeval framework. we should note that these datasets either (a) already existed prior to this publication or (b) are small extensions of existing datasets. an overview of all the datasets can be found in ## discussion having a good mean rank score is not the only thing that matters when choosing a model for a given task. model size, inference speed and whether the model has publicly available weights are all important factors to consider. for this reason, we also include these metadata in the leaderboard, and we encourage the community to consider these factors when choosing a model for a given task. some of the datasets in the benchmark are translations of american datasets, which we acknowledge is not ideal and encourage the development of gold-standard replacements of these. this concerns the dutch question answering dataset, which is machine translated, as well as the danish and swedish question answering datasets, where the questions and answers have been manually translated. manual translations are typically better than machine translations, but it nevertheless means that the content is biased towards questions pertinent to the american context. some datasets are furthermore missing. this concerns icelandic and faroese sentiment analysis, as well as faroese question answering. efforts are currently underway to remedy this. lastly, we note that the english sentiment classification dataset sst5 is the only dataset where generative models perform substantially better than encoder models. we speculate that this is either due to the dataset simply being significantly easier than the others, or that the test data has leaked into the pretraining datasets of the generative models. the dataset is part of the flan collection @xcite , which is for instance included in the dolma dataset @xcite , which is used to pretrain the olmo model @xcite , being one of the generative models that is performing very well on this dataset. leakage is therefore possible, and we encourage the community to investigate this further. ## conclusion we have extended the scandeval benchmark to include the evaluation of decoder models, as well as including three new languages: german, dutch and english. from the analysis of the corresponding results we found that encoder models can achieve significantly better nlu performance than decoder models despite having orders of magnitude fewer parameters, but that this varies between languages. we have also shown that being generative is strongly correlated with both good question answering performance and poor performance for named entity recognition and linguistic acceptability. our analysis showed that the "path" from the worst to the best-performing models in the umap space is different for encoder and decoder models, indicating an architecture-specific task-preference.