Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles (2025)

Abstract: We introduce a dataset of high-quality human-authored summaries of news articles in Norwegian. The dataset is intended for benchmarking of the abstractive summarisation capabilities of generative language models. Each document in the dataset is provided with three different candidate gold-standard summaries written by native Norwegian speakers and all summaries are provided in both of the written variants of Norwegian – Bokmål and Nynorsk. The paper describes details on the data creation effort as well as an evaluation of existing open LLMs for Norwegian on the dataset. We also provide insights from a manual human evaluation, comparing human-authored to model generated summaries. Our results indicate that the dataset provides a challenging LLM benchmark for Norwegian summarisation capabilities.

## introduction one of the key practical use cases of large language models (llms), is to generate condensed summaries of texts. several news publishers already include llm-generated summaries as part of the news stories they publish. evaluating such generated summaries, however, remains a challenge. for norwegian, one important reason for this is the lack of gold-standard summaries to compare to. the current paper introduces a new and open dataset of high-quality human-authored summaries of news articles in norwegian, covering both of the official written variants; bokmål (bm) and nynorsk (nn). aiming to make benchmarking as robust as possible, each document in the dataset is provided with three different candidate gold-standard summaries (for each variant, bm and nn, resulting in six summaries in total for each news article). the remainder of the paper is structured as follows. we first describe the creation of the human-authored summaries, including the underlying data sources, the annotator guidelines, and corpus statistics. we then move on to describe a first set of experiments with using pre-trained llms to generate summaries, and then evaluate them using our new dataset. importantly, we here also present the methodology and framework we use, including factors like prompts and metrics. we thereafter discuss in detail the setup and results of our manual human evaluation. ## related work summarisation datasets are foundational for advancing the development of techniques for automatic summarisation, as well as for benchmarking llms. there are various approaches developed to address diverse summarisation challenges, along with influential datasets to benchmark both extractive and abstractive methods @xcite . most works on benchmark datasets have been done for english, and we here mention some of the works that focus on summarising news articles. the cnn/daily mail dataset @xcite is one of such influential works. this dataset was created for the task of reading comprehension, but is widely used for summarisationrelated tasks. the dataset consists of news articles accompanied by a set of bullet points representing (abstractive) summaries. subsequent works have focused on creating resources for various domains, contexts, and summarisation styles. for instance, gigaword @xcite is extracted from the gigaword news corpus and contains sentences paired with short summaries (headlines). this is also an abstractive dataset enabling sentencelevel summarisation. it has however been criticised for only including headlines instead of full summaries @xcite . the extreme summarisation (xsum) dataset @xcite was also created from news articles, sourced from bbc. each article in this dataset is paired with a one-sentence summary representing a concise and abstractive summary. the cnn-corpus @xcite ) contains news articles from cnn paired with highlights and gold-standard abstractive summaries. however, the corpus is mostly used for extractive summarisation tasks @xcite . efforts have also been made for multi-document summarisation, such as multi-news @xcite , which contains relatively long summaries of multi-news articles covering the same topic. resources for news summarisation in norwegian are notably scarce. some efforts to introduce summarization datasets in norwegian have relied on machine translation, e.g. based on the cnn/dailymail data @xcite . however, failing to adequately capture nuances of the target language, as machine translation may produce non-idiomatic and non-natural-sounding language. another concern is that, being based on english sources, the original texts are typically not geared towards issues of primary salience to a norwegian context (whether socially, politically, geographically, or otherwise), which is unfortunate if the goal is to benchmark norwegian llms. to our knowledge, no freely available, manually curated summarization dataset, created from scratch for norwegian news data exists, making this work a valuable contribution to advancing research in this field. ## human authored summaries data sources we use a subset of the news articles in the norwegian event extraction dataset eden @xcite as the data source for summarisation. eden contains articles in bm only, and because creating summaries based on news articles is a time and effort intensive task, we here only make use of the dev and test splits of eden, which respectively contain 30 and 33 news articles. eden was chosen due to its high-quality, as it comprises news articles from the norwegian dependency treebank @xcite , and is a richly annotated dataset covering event triggers and arguments @xcite , named entities @xcite , morphosyntactic annotation, and co-reference information @xcite . annotators we hired three annotators with strong academic backgrounds related to journalism, all norwegian native speakers. the annotators were fairly compensated following an hourly contract, and were hired for a period of 6 months. all annotators have a background in media science or journalism. the first annotator, has a bachelor's degree in media and communication science, and has worked as a freelance journalist. the second annotator has a bachelor's degree in journalism, and was finishing up a master's degree in investigative journalism while doing an internship in a leading norwegian news broadcasting company. the third annotator, a journalism student, who also worked part-time as a journalist in a local norwegian newspaper. all hired annotators have experience writing news articles, including the identification of key information that should be selected to write the article. as the task was to create natural-sounding summaries that preserved the original meaning of the news articles, we believe that these annotators can be referred to as domain experts. in addition, as we wanted the summaries to be as naturalsounding as possible, we asked the annotators to write in their preferred variant of norwegian. this has resulted in two annotators writing in bm, and one annotator writing in nn. guidelines the annotators received a detailed set of guidelines outlining the steps to follow when authoring the summaries. the guidelines were inspired by concrete prompts, shared with us, and which were used to automatically generate summaries of news articles by one of the biggest media companies in norway. we asked the annotators to write summaries that reflect the main content of the news articles, but without providing strong limitations to their language use or formulations. we aimed to create summaries that are as natural-sounding as possible, and as diverse as possible. each annotator was free to write their own summaries, without consulting or discussing details about the content of the summaries. however, we provided the annotators with an example consisting of a news article news article mer frukt, men mindre norsk | forbruket økt med 20 prosent på ti år. forbruket av frukt og grønt har økt med over 20 prosent i løpet av de siste ti årene. men den norske produksjonen faller. hele veksten og mer til av frukt og grønt kommer fra import. den norske produksjonen har nemlig falt med 10 prosent siden 1998, skriver nationen. ifølge landbruks-og matminister lars peder brekk (sp) må de norske kjedene bli flinkere til å samarbeide med norske produsenter og bøndene må bli flinkere til å produsere det kundene vil ha. summary 1 summary 2 summary 3 bokmål forbruket av frukt og grønt har økt med 20 prosent på ti år. hele veksten og mer til av frukt og grønt kommer fra import da den norske produksjonen har falt med 10 prosent siden 1998, skriver nationen. de norske kjedene må bli flinkere til å samarbeide med norske produsenter og bøndene må bli flinkere til å produsere det kundene vil ha ifølge landbruks-og matminister lars peder brekke (sp). nordmenn kjøper mer frukt og grønnsaker, samtidig som de norske bøndene produserer mindre. frukt-og grønt-forbruket har økt med over 20 prosent de 10 siste årene. den norske produksjonen har falt med 10 prosent siden 1998. import av varer dekker den økte etterspørselen i det norske markedet. forbruket av frukt og grønt har økt med 20 prosent i norge de siste ti årene. likevel falt den norske eksporten, og veksten kommer fra stadig mer import. siden 1998 har den norske produksjonen falt med 10 prosent, opplyser nationen. landbruksministeren sier at kjedene må bli flinkere til å samarbeide med norske produsenter, og at bøndene i større grad må produsere det kundene ønsker. nynorsk forbruket av frukt og grønt har auka med 20 prosent på ti år. heile veksten og meir av frukt og grønt kjem frå import då den norske produksjonen har falle med 10 prosent sidan 1998, skriv nationen. dei norske kjedane må bli flinkare til å samarbeide med norske produsenter og bøndene må bli flinkare til å produsera det kundene vil ha, ifølge landbruks-og matminister lars peder brekke (sp). nordmenn kjøper meir frukt og grønnsaker, samtidig som dei norske bøndene produserer mindre. frukt-og grønt-forbruket har auka med over 20 prosent dei 10 siste åra. den norske produksjonen har gått ned med 10 prosent sidan 1998. import av varer dekker den auka etterspørselen i den norske marknaden. forbruket av frukt og grønt har auka med 20 prosent i noreg dei siste ti åra. likevel fell den norske eksporten, og veksten kjem frå meir og meir import. sidan 1998 har nemleg den norske produksjonen falle med 10 prosent, opplyser nationen. landbruksministeren seier at kjedene må bli flinkare til å samarbeida med norske produsentar, og at bøndene må i større grad produsera kva kundane ynskjer. we intentionally decided to keep the annotation guidelines simple to give annotators the freedom to write in a natural and authentic style. rather than imposing strict constraints, we provided them with general and broad instructions on the importance of maintaining journalistic integrity while clearly, precisely, and concisely creating an informative summary. we believe that this flexibility allowed annotators to create more natural and engaging summaries. our choice of enforcing summaries formatted as bullet-points was in part based on how news outlets present machine-generated summaries in the norwegian news. but also because we planned to perform a human evaluation where human-authored summaries will be compared to machine-generated summaries. see section 6 for more details about this analysis. ## evaluation design in the following, we illustrate the use of our summarisation dataset as an evaluation benchmark for a range of openly available norwegian and multilingual llms. models we evaluate nine pretrained transformer llms as our baselines: norwai-mistral-7b 2 , nora.llm (norbloom-7b-scratch 3 , normistral-7b-scratch 4 , and normistral-7bwarm 5 ; samuel et al., 2025), norwai-llama2-7b 6 , viking-7b 7 , viking-13b 8 , mistral-7b-v.01 9 @xcite , and falcon-7b 10 (almazrouei et al., 2023). all the llms' weights are taken from the transformers library @xcite . setup we conduct a zero-shot evaluation of the previously mentioned llms using noreval 11 , an open-source framework for evaluating norwegian generative llms. we integrate our dataset into noreval together with 12 diverse prompts written by norwegian native speakers, who are authors of this paper. performance metrics we measure the performance using standard summarisation evaluation metrics: rouge-l @xcite and bertscore @xcite . our result aggregation procedure accounts for prompt sensitivity @xcite and includes two steps: (i) for each prompt, we compute the maximum performance scores between the llm's output and each of three human-written references (our human-authored summaries); (ii) we then maximize the bertscore across all prompts and average the resulting rouge-l and bertscore values over all bm/nn examples. ## evaluation results ## human evaluation in addition to model and metric-based evaluations, we conducted a manual evaluation. for this purpose, a research assistant was hired to develop an interface where evaluators were shown a news article, followed by two summaries beneath it. an example of this simple interface is shown in figure 2 . the volunteer evaluators were asked to choose their preferred summary from a selection of two summaries: one human-authored and one generated by a model. however, the evaluators were not aware of the provenance of each summary. to ensure that evaluators rank summaries consistently, we provided them with a set of very simple criteria inspired by evaluations presented in @xcite : we also asked the evaluators to prioritise these criteria in the following order: @xmath0, with relevance being the most important and fluency the least. this approach was designed to assess the quality of the summaries based on the primary functions of summarisation: accurately and concisely conveying essential content. the prioritisation we chose reflects a deliberate emphasis on accuracy and factuality over style. the link to this evaluation interface was shared with volunteer colleagues, resulting in a total of 146 responses. in 138 cases, evaluators preferred the human-authored summaries, while only 8 responses favoured a machine-generated summary. these preferred machine-generated summaries were produced by the three models viking-13b (4 of the preferred summaries), norbloom-7b-scratch (2 of the preferred summaries), and normistral-7b-warm (2 of the preferred summaries), using prompt nr. 1 (bm) and prompt nr. 2 (bm) in similarly to the results in several issues were identified during the human evaluation of summaries. these were primarily related to those generated by the models. we give a summary of the types of errors that commonly appeared in what follows. issues related to relevance the generated summaries often reproduce the initial part of the original article, not including important information presented later, and sometimes even cutting off mid-sentence. some summaries were direct copypaste of the original article, or were too lengthy, and occasionally repeating (parts of) the prompts (e.g. "skriv en oppsummering av følgende artikkel med kun noen få punkter: tilbake til hverdagen | helse. vandrehall [. . . ]", eng: write a summary of the following article in just a few points: back to everyday life | health. walking hall [. . . ]). some other summaries were too short, providing incomplete contexts or unnaturalsounding sentences. issues related to consistency generally, evaluators reported that the summaries were consistent with the source material. however, some summaries did exhibit repetitions of phrases. minor but significant alterations in the texts, like adding or omitting words, were also observed. in some instances, the model-generated summaries invented quotes (e.g. a citation in the summary that did not occur in the original news text "-jeg er veldig glad for at jeg har fått et nytt hjerte, sier per arne olsen til tønsbergs blad." (eng: '-i am very happy that i have received a new heart, says per arne olsen to tønsbergs blad."). however, a simple internet search led us to finding a similar quote in another news article which seemingly the model had access to during training), or confused entities (e.g., mixing between bill and hillary clinton when mentioned jointly in a news article). issues related to fluency similarly to what we already have mentioned, despite fluency being largely maintained, certain summaries repeated identical or similar sentences continuously (more than 10 times). additionally, in some cases we observed missing function words (e.g. the function word "av" (eng: by) in the sentence "malis statsminister cheick modibo diarra har gått av etter å ha blitt pågrepet soldater" (eng: mali's prime minister cheick modibo diarra has resigned after being arrested soldiers) not being included in the same sentence in the generated summary.) ## conclusion and outlook this paper introduces a novel dataset of humanauthored summaries of norwegian news articles for benchmarking abstractive summarisation. our dataset is of high quality and provides for each news article a set of diverse summaries written in both norwegian varieties bokmål and nynorsk. through comprehensive evaluations using human evaluators and generative models, we have demonstrated the robustness and complexity of this benchmark. as this is the first freely available humanauthored norwegian summarisation datasets, we believe that the impact it will have on benchmarking current and future llms is considerable. looking ahead, we see several avenues for developing models that leverage the particularities of this dataset to build more robust summarisation techniques. this dataset allows us to compare the output of generative models to a distinct set of human-authored summaries, which will allow us to generate more naturally-sounding summaries.