Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024)

Abstract: The natural language processing field has been evolving around language models for the past few years, from the usage of n-gram language models for re-ranking, to transfer learning with encoder-only (BERT-like) language models, and finally to large language models (LLMs) as general solvers. LLMs are dominated by the decoder-only type, and they are popular for their efficacy in numerous tasks. LLMs are regarded as having strong comprehension abilities and strong capabilities to solve new unseen tasks. As such, people may quickly assume that decoder-only LLMs always perform better than the encoder-only ones, especially for understanding word meaning. In this paper, we demonstrate that decoder-only LLMs perform worse on word meaning comprehension than an encoder-only language model that has vastly fewer parameters.

## introduction large language models (llms) are highly effective tools for solving different kinds of problems in natural language processing @xcite , computer vision @xcite , robotics @xcite , and more. due to their fascinating abilities to solve a myriad of tasks, large language models, which are dominated by the decoder-only type, are often considered general problem solvers @xcite . moreover, large language models are able to perform tasks outside of what they were trained on with few to no examples, a phenomenon referred to as emergent abilities @xcite . large language models have been shown to have strong comprehension abilities @xcite . as such, it is natural to think that they are the best for lexical semantics. however, @xcite reported that decoder-only language models struggle with understanding more nuanced contextual features. this motivates us to investigate the semantic understanding of decoder-only language models compared to the encoder-only ones. we evaluate the semantic understanding capability of language models through the word sense disambiguation (wsd) and word-in-context (wic) tasks. word sense disambiguation is the task to determine which sense of a word is meant in a particular context, while the word-in-context task is to identify whether the word senses of the same word in two sentences are the same or not (figure 1 ). we investigate both open-source decoder-only models, mistral @xcite and llama 2 @xcite , and a closed-source model, gpt-4 foot_0 (openai, 2023). we compare the results against a strong encoder-only model, deberta-v3-large @xcite . ## results on the wic task, the zero-shot scores of mistral and llama are close to random chance, probably because the models do not understand the expected output without any example ( mistral 7b was reported to outperform llama 7b and 13b across a wide range of benchmarks @xcite ), but we find that llama performs better in almost all experimental settings. on the other hand, gpt-4 in the zero-shot setting outperforms both mistral 7b and llama 13b in their few-shot settings. gpt-4 performs exceptionally well in the zero-shot setting of the wsd task but only gains very slight improvement with in-context learning. as gpt is a closed-source model, information on its model training procedure and data is not accessible. we cannot rule out the possibility that gpt may have been trained on the wsd task or with wsd training and test data. on the wic task, fine-tuned llama 2 performs close to deberta-v3-large. however, the score difference is still large when compared to the bigger deberta 1.5b, which achieves an accuracy of 76.4 on the wic task @xcite . ## related work prior work has reported performance comparisons of decoder-only models with encoder-only models on different tasks. @xcite investigate the performance of decoder-based llms on financial text analytics tasks and report that gpt-4 with incontext learning has comparable performance to fine-tuned encoder-only models on financial sentiment analysis and lower performance on financial headline classification and relation extraction. @xcite investigate llama 2 and gpt-4 on named entity recognition, political ideology prediction, and misinformation detection. they also find that gpt-4 can achieve comparable performance on some of the tasks and perform worse on others. these works support our findings that llms still have limited capabilities on some downstream tasks, but they have not investigated llms' capabilities in understanding word meaning. ## conclusion and future work in this paper, we demonstrate that decoder-only llms perform worse on word meaning comprehension than an encoder-only language model with vastly fewer parameters. we report the performance of mistral 7b, llama 2 13b, and gpt-4 on the wsd and wic tasks and show that deberta-v3-large with vastly fewer parameters outperforms all aforementioned decoder-only models on both tasks. we discuss the importance of the encoder in understanding word meaning by running the same experiment on an encoder-decoder model, flan-t5-xl, and find that it also outperforms the decoder-only models. this work provides a concrete task on which encoder-only language models can still outperform decoder-only language models, but it is not meant as a study on neural network architecture. future research can explore the optimal architecture and the roles of the encoder and decoder in natural language understanding. additionally, investigating methods to enhance the capabilities of decoderonly models in comprehending word meaning is also a valuable research direction.