Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance (2025)

Abstract: The increasing demand for efficient summarization tools in resource-constrained environments highlights the need for effective solutions. While large language models (LLMs) deliver superior summarization quality, their high computational resource requirements limit practical use applications. In contrast, small language models (SLMs) present a more accessible alternative, capable of real-time summarization on edge devices. However, their summarization capabilities and comparative performance against LLMs remain underexplored. This paper addresses this gap by presenting a comprehensive evaluation of 19 SLMs for news summarization across 2,000 news samples, focusing on relevance, coherence, factual consistency, and summary length. Our findings reveal significant variations in SLM performance, with top-performing models such as Phi3-Mini and Llama3.2-3B-Ins achieving results comparable to those of 70B LLMs while generating more concise summaries. Notably, SLMs are better suited for simple prompts, as overly complex prompts may lead to a decline in summary quality. Additionally, our analysis indicates that instruction tuning does not consistently enhance the news summarization capabilities of SLMs. This research not only contributes to the understanding of SLMs but also provides practical insights for researchers seeking efficient summarization solutions that balance performance and resource use.

## introduction automatic text summarization is a fundamental challenge in natural language processing (nlp) that plays a crucial role in various applications, including search engine optimization @xcite , financial forecasting @xcite , and public sentiment analysis @xcite . the ability to distill large volumes of information into concise summaries is particularly * corresponding authors important in today's fast-paced digital environment, where users often seek quick insights from news articles. recent advancements in large language models (llms) such as gpt-3 @xcite , opt @xcite , and llama2 @xcite have significantly enhanced the quality and coherence of generated summaries compared to traditional models @xcite @xcite @xcite . however, the hundreds of billions of parameters in these llms require substantial computational and storage resources. for example, deploying the llama2-70b model at fp16 precision typically necessitates two nvidia a100 gpus with 80gb of memory each. this means that only organizations with significant computing power can offer llm services, raising concerns about service stability and data privacy. as a result, general-purpose small language models (slms) have emerged as a potential alternative, as shown in figure 1 . they share the same decoderonly architecture as llms but have fewer than 4b parameters 1 @xcite @xcite . same as the llms, they generate specified content based on prompts. slms are designed to run efficiently on edge devices like smartphones and personal computers, making them more accessible. this accessibility makes slms an attractive option for applications requiring quick, stable, and low-cost solutions that also protect user privacy, such as summarizing news articles for mobile users or enhancing data processing privacy in sensitive environments. despite the promising capabilities of slms, their specific performance in the domain of news summarization remains underexplored. previous studies, such as tiny titans @xcite , have evaluated specifically fine-tuned slms in meeting summarization. the adopted rouge metric @xcite only reflects the quality in terms of relevance. in addition, broader evaluations of slms across various tasks have been evaluated but omitted news summarization @xcite , highlighting the gap in understanding slm in this area. this paper aims to address these gaps by conducting a comprehensive evaluation of pre-trained general-purpose slms in news summarization, a process that abstracts news articles into shorter versions while preserving key information. we seek to answer the following research questions: first, how do different slms compare to each other, and how do they compare to llms? we benchmark 19 slms on 2,000 news samples foot_1 . our findings reveal that the news summarization capabilities of different slms vary significantly as shown in figure 1 . and the best-performing slms can generate news summaries comparable in quality to those produced by llms while producing shorter summaries. notably, phi3-mini and llama3.2-3b-ins emerge as the top performers. second, what is the impact of prompt complexity on the performance of slms? we compare the impact of prompts with varying levels of detail on the quality of the summaries. our findings indicate that prompt engineering has a limited impact on enhancing summarization abilities, with even complex prompts potentially degrading model performance. thus, simple prompts suffice in practical applications. third, how does instruction tuning affect the summarization capabilities of different slms? we find that the effects of instruction tuning on slms vary significantly. while the llama3.2 series models show substantial improvements after instruction tuning, which aligns with existing works @xcite , models like qwen2 and internlm2 exhibit minimal changes. the rest of the paper is organized as follows: section 2 covers the background and related work, section 3 introduces reference summary generation, section 4 illustrates the benchmark design, section 5 presents presents the benchmarking results of slms, section 6 analyzes the influencing factors on slm summarization, and section 7 concludes our findings. ## llm-augmented reference-based evaluation as the reference-based method offers higher efficiency, better reproducibility, and lower cost compared to human and llm evaluation @xcite , we use the reference-based method for large-scale slm evaluation. however, some existing research @xcite indicates that the poor quality of heuristic reference summaries in current datasets weakens their correlation with human preferences. moreover, @xcite find that using high-quality summaries as references can significantly improve alignment with human preferences. thus, we use llm-generated summaries as references in our slm evaluation instead of relying on original dataset references. llm-generated reference summaries offer two key advantages. first, they have higher quality than original heuristic summaries; @xcite find llms averaged 4.5 on a five-point scale, while original references scored only 3.6. additionally, @xcite find that llm-generated summaries are preferred to those written by freelance authors by up to 84% of the time. second, llms produce high-quality summaries quickly, generating hundreds in a few hours, which benefits largescale news evaluations. in contrast, a freelance writer typically takes about 15 minutes for a single summary @xcite . we also verify whether the llm-generated reference summaries can improve the effectiveness of reference-based evaluation on three human evaluation datasets @xcite in ## benchmark design in this section, we first introduce the dataset for news summarization evaluation, followed by the slm details, evaluation metrics, and reference summary generation. ## influencing factor analysis 6.1 prompt design prompt engineering has demonstrated significant power in many tasks, helping to improve the output quality of llms @xcite . therefore, we use different prompt templates shown in figure 3 to analyze the impact of prompt engineering on bertscore scores, factual consistency, and summary length. the instructions become more detailed from prompt 1 to prompt 3. overall, the improvement of prompt design on the quality of slm summaries is limited. for slms that already perform well with simple prompts, they can capture the key points without the need for complex prompts. in contrast, slms that perform poorly may struggle with the logical relationships in complex prompts, leading to lengthy summaries. for practical applications, we recommend using simple and clear prompts when deploying slms for news summarization. ## conclusion this paper presents the comprehensive evaluation of slms for news summarization, comparing 19 models across diverse datasets. our results demonstrate that top-performing models like phi3-mini and llama3.2-3b-ins can match the performance of larger 70b llms while producing shorter, more concise summaries. these findings highlight the potential of slms for real-world applications, particularly in resource-constrained environments. further exploration shows that simple prompts are more effective for slm summarization, while instruction tuning provides inconsistent benefits, necessitating further research.