CNNS um: Exploring Long-Context Summarization with Large Language Models in C hinese Novels (2025)

Abstract: Large language models (LLMs) have been well-researched in various long-context tasks. However, the scarcity of long-context summarization datasets hinders progress in this area. To address this, we introduce CNNSum, a multi-scale long-context summarization benchmark based on Chinese novels, featuring human-driven annotations across four subsets totaling 695 samples, with lengths ranging from 16k to 128k. We benchmark numerous LLMs and conduct detailed human assessments to summarize abnormal output types. Furthermore, we extensively explore how to improve long-context summarization. In our study: (1) Advanced LLMs may generate much subjective commentary, leading to vague summaries. (2) Currently, long-context summarization mainly relies on memory ability. The advantages of Large LLMs are hard to utilize, thus small LLMs are more cost-effective. (3) Different prompt types paired with various version models may cause large performance gaps. In further fine-tuning, these can be mitigated, and the Base version models perform better. (4) LLMs with RoPE-base scaled exhibit strong extrapolation potential; using short-context data can significantly improve long-context summarization performance. However, further applying other interpolation methods requires careful selection. (5) CNNSum provides more reliable evaluation results than other benchmarks. We release CNNSum to advance future research.

## introduction long-context large language models are advancing rapidly @xcite , making complex long-context tasks that were once considered difficult achieve revolutionary progress @xcite . however, long-context summarization tasks progress slowly, which is crucial for assessing long-context ability and forms the basis for complex contextual understanding @xcite @xcite . @xcite declare that traditional summarization research is dead, but creating new high-quality datasets remains essential. unlike most tasks, summarization is more subjective, without a gold standard. and obtaining high-quality data is challenging for long texts, which requires a global context understanding and memory, and even human experts struggle to annotate directly @xcite . recent studies on llms summarization @xcite @xcite are still based on previous short datasets @xcite . @xcite proposes unlimited-input transformers based on summarization task, using only booksum @xcite as sufficiently long data, which remains the latest book-length dataset. the maximum length of newer chinese long summarization datasets @xcite is only 6k. @xcite study book-length summarization, but both are expensive in annotating their own datasets to avoid leakage risks. most studies focus on hallucination-related issues evaluations. however, implementing these evaluations holds limited significance if models cannot work properly. although the 128k context length has been common, llms exhibit large performance degradation on various tasks as sequence length increases @xcite . in this study, we indeed find that outputs may be disorganized, meaningless, or fail to follow the instructions. the core is how llms handle long-context summarization tasks. due to a lack of datasets, there is still insufficient systematic research and guidance. despite many long-context llms benchmarks emerging @xcite @xcite @xcite , their summarization tasks still have considerable room for improvement in terms of construction. regardless of language and domain, we summarize the shortcomings: (1) building on old datasets @xcite @xcite @xcite leads to high leakage risks @xcite . (2) data volume is small, maybe just dozens. @xcite the average and maximum lengths are short, typically under 16k. (4) lacking multi-scale length-based subsets limits evaluation in varying contexts, and our experiments indicate a wide range of mixed lengths may lead to misleading results. (5) annotations are either collected from the web, carrying high leakage risk and low quality, or synthesized by llms and may have various errors @xcite @xcite . each benchmark mentioned above has at least two of these shortcomings. to better explore long-context summarization with llms, we introduce chinese novel summarization (cnnsum), a new multi-scale dataset that improves all the above shortcomings. based on a newly collected corpus, we built four subsets: l(16k), xl(32k), 2xl(64k), and 3xl(128k), totaling 695 samples. the annotations are completed by human experts with assistance from llms. since prompt formats greatly impact llms @xcite and beginning and ending tokens are crucial in long contexts @xcite , we define two prompt types for baselines: instruction at the beginning (prompt-ib) and instruction at the end (prompt-ie), as shown in appendix i.1. although rouge-l @xcite remains the primary metric in most benchmarks, it differs significantly from human preferences. even llms are not yet reliable enough to replace humans @xcite . while referencing automatic metrics, we conduct fine-grained inspection and analysis of the outputs to categorize error types. our observations are as follows: @xcite prefers more subjective narration leading to vague summaries, so focusing on objective plots remains the key. @xcite the two prompts can result in a significant output quality gap across various llms. @xcite the chat or instruction version may harm the base model's long-context summarization ability. @xcite the large llms struggle to fully leverage their advantages in reasoning and comprehension, while small llms are more cost-effective. the abnormal behaviors are usually related to training configurations of llms; we further conduct extensive fine-tuning experiments. the llms may have acquired longer-context abilities during pre-training @xcite , requiring only activation. consequently, we concatenate short summarization data into longer ones for training. our findings indicate that: (1) the performance gaps caused by prompts can be mitigated, allowing models to fully perform. (2) the base models are better for further fine-tuning and extrapolation than the instruction or chat version. in summary, we introduce cnnsum, an entirely new long-context chinese novel summarization benchmark, which is significantly superior in design and construction compared to existing datasets. we carry out comprehensive explorations of longcontext summarization tasks for llms, providing guiding insights for future research in this field. ## experiments 4.1 baselines chinese has one of the largest corpora, but many advanced llms still support it poorly, as shown in appendix c.1. such as claude3.5 @xcite and gpt-4o cannot process 3xl. we select models as follows; details are in appendix c.2. commercial models (1) openai flagship, gpt-4o-2024-08-06 (openai, 2024b) and the lite version gpt-4o-mini-2024-07-18 (openai, 2024a). open-source models (1) yi @xcite and yi-1.5 series. (2) qwen1.5, qwen2 @xcite , and qwen2.5 @xcite series. (3) chatglm3-6b-128k @xcite and special version glm4-9b-chat-1m @xcite . ( 4 ) internlm2.5-20b @xcite series and special version internlm2.5-7b-chat-1m. (5) llama3.1 series @xcite . (6) ministral-8b-instruct-2410 (ai, 2023). (7) lwm-text-1m @xcite . ## conclusion we present cnnsum, a new long-context chinese novel summarization benchmark dataset. it has four subsets ranging from l (16k) to 3xl (128k), comprising 695 samples with human annotations. we conduct extensive experiments on cnnsum to investigate the long-context summarization. we conduct a fine-grained human assessment and analysis of the model outputs, revealing the deficiencies and issues of current llms in handling longcontext summarization. we further explore how to improve the performance of llms through finetuning with concatenated short-context data and confirm that cnnsum can offer more reliable evaluation results. we hope cnnsum advances future research and provides valuable insights.