Are Small Language Models Ready to Compete with Large Language Models for Practical Applications? (2025)

Abstract: The rapid rise of Language Models (LMs) has expanded their use in several applications. Yet, due to constraints of model size, associated cost, or proprietary restrictions, utilizing state-of-the-art (SOTA) LLMs is not always feasible. With open, smaller LMs emerging, more applications can leverage their capabilities, but selecting the right LM can be challenging as smaller LMs don’t perform well universally. This work tries to bridge this gap by proposing a framework to experimentally evaluate small, open LMs in practical settings through measuring semantic correctness of outputs across three practical aspects: task types, application domains and reasoning types, using diverse prompt styles. It also conducts an in-depth comparison of 10 small, open LMs to identify best LM and prompt style depending on specific application requirement using the proposed framework. We also show that if selected appropriately, they can outperform SOTA LLMs like DeepSeek-v2, GPT-4o-mini, Gemini-1.5-Pro, and even compete with GPT-4o.

## introduction the field of nlp has advanced significantly with the rapid development of language models (lms) @xcite @xcite @xcite , which has expanded their use across numerous types like title generation (keles and bayraklı, 2024), data exploration @xcite , dialogue act recognition @xcite ; domains like economics & finance @xcite , politics @xcite , news @xcite ; and reasoning types @xcite like analog-ical @xcite and @xcite reasoning. despite the growing variety of lms, their usage in downstream applications is heavily skewed towards limited ones. analyzing around 50 papers from 2024, we found that while 82.3% of methods utilized gpt-family lms, only 41.1% used llama variants, and less than 11.8% experimented with other alternatives like mistral @xcite and falcon @xcite . some studies also report issues like garbage output and hallucinations @xcite , but domain experts often lack the tools to address them effectively through informed lm choice, or correct ways to prompt them. even the methods that experiment with multiple lms often select models without a strong motivation @xcite . apart from performance, many of the new lms are smaller in size, and openly available. despite the undeniable success of large, proprietary lms like @xcite and llama-2 70b @xcite , their inaccessibility due to limited api access, high costs @xcite , concerns around data privacy (for gpt), and massive computational demands @xcite (for llama) pose significant barriers of usage. small, open lms can navigate around those, and also provide additional benefits like ondevice usage, faster inference time, data privacy, easier compliance and security management, and low-cost maintainance. for many practitioners -especially those in research, startups, or sectors with limited resources or high security risk -leveraging these presents an appealing alternative for functional, financial, or business reasons. but these new, small, lms vary a lot in terms of training data, pre-training strategies, and architectural decisions. additionally, they may not perform globally well like sota llms due to limitations of scale @xcite . utilization strategies of lms in inference pipelines can also differ, like zero-shot usage, customizing pre-trained models (e.g., fine-tuning @xcite ), using in-context learning @xcite , prompt engineering @xcite . writing effective prompts also requires time and domain expertise. so, users need to conduct thorough analysis before choosing the right lm and usage strategy within constraints of time, money, computational resources, which is a complicated task. although technical reports of some lms @xcite provide some insights, not all of them capture real-world, practical scenarios. therefore, there is a need for a comprehensive practical evaluation framework which can enable determining capabilities of lms in multiple practical applications, and effective ways to prompt them. to bridge this gap, we propose a comprehensive framework for evaluating lms in practical settings along three aspects: task types, application domains, and reasoning types. for each aspect, we select 12, 12, and 10 entities in english, grouping similar ones (e.g., @xcite . this three-tier structure (aspect, group, entity) helps identifying patterns in lm capabilities across multiple levels. using super-natural instructions @xcite , a meta-dataset encompassing various nlp benchmarks, we evaluate lms on task instances within this framework. lm usage strategies vary significantly -ranging from fine-tuning @xcite , peft @xcite or direct usage with/without prompt engineering. thus, we assess semantic correctness of outputs as an indicator of lms' inherent abilities, evaluating five pre-trained and five instruction-tuned (it) @xcite models across eight prompt styles. our results show that with careful selection, impact of scale can be reduced. correctly chosen small, open lm can rival and even outperform models like gpt-4o-mini, @xcite , @xcite , and gemini-1.5-pro @xcite , while providing additional benefits. we also evaluate lms with paraphrases of task definitions to show that results are robust against dataset-induced biases. in this work, we aim to address these research questions: (i) can small, open lms compete with large, proprietary lms in practical usage? (ii) what can be an exhaustive evaluation framework to conduct this analysis? (iii) for different application needs, how do current best small, open lms perform in comparison, and which lm is the best choice? (iv) what type of prompt style should be used to extract best results from these lms? consistent with figure 1 , we make the following key contributions: ## evaluation framework we begin with describing our evaluation framework discussing dataset, prompt styles, selection process of aspects, evaluation metrics and experiments. ## experiments and results we use all the prompt styles with each of the task instance, do a forward pass on the lm, and decode the output using greedy decoding, which is evaluated with available references. we used greedy as it's reproducible, also other sampling techniques @xcite didn't give any improvement (refer appendix e). the following subsections discusses findings. ## high-level takeaways we find that recent, open and small-scale language models (lms) are very effective. detailed recommendations on lms and their performance trends in different groups and entities are discussed in for the lms we experimented with, among pretrained models, we recommend using gemma-2b and falcon-2-11b based on different aspects and entities, but sometimes, gemma-7b, llama-3-8b can be great choices (also detailed in appendix b). the performance of pre-trained models can be taken as a measure of their knowledge of different use-cases. based on other factors like availability, compliance, size, right lm can be selected and customized as needed. limitations of some pre-trained models are discussed in appendix f.2. for it models, mistral-7b-i is a clear best in all aspects, and gemma-2b-i and smollm-1.7b-i come second in most cases. since these models are it, they can be used directly with chat-style description and examples. we recommend a model in these three (and other models), based on other factors like size, licensing, etc. some qualitative outputs of mistral-7b-i are given in appendix f.1. we also study the performance trade-off for multiple prompt styles and recommend the best one for these models. as noted in section 3.6, having a chat-style task description to guide the lm is recommended. but, having more in-context examples is not always better, and considering use-case and lms, the right number of example can vary. the models are also robust to changes in task definitions, if it can provide all (or most) information to complete the task. they are also reasonably robust to subtle intended/unintended incorrectness in definitions, which is analyzed in appendix d.4. in appendix b, we also compare and show that the small lms even outperform deepseek-v2 (ds-2) (deepseek-ai, 2024) in many categories. ## conclusion we identify some limitations of using sota, proprietary llms and show that open lms with 1.7b-11b parameters can be effective for applications. we create a three-tier evaluation framework and analyze semantic correctness of output of 10 lms across multiple hierarchical umbrellas. using this framework, we demonstrate that while these models don't work best in every scenario, if selected properly, they are effective and can compete with and outperform models like gemini-1.5-pro, gpt-4o-mini and gpt-4o despite being 10-20 folds smaller in size. we also provide a guide in appendix a on how one can this work to select an lm for one's specific needs. we hope that our contributions will enable the community to make a confident shift towards considering using these small, open lms for their need. limitations of lms: there are many lms proposed by the research community, and it is not practically feasible to experiment with all of them. further, the definition of a small lm is also relative. we selected the lms based on the ones which have come out recently and promise strong capabilities. although we capped our lms at 11b parameters, we didn't find the performance to be a direct function of number of parameters, so we believe this decision should not have too drastic effects. we also didn't evaluate biases @xcite and other factors other than semantic correctness of generated results of these models, but the models' technical reports @xcite @xcite @xcite provide more details on those.