The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective (2025)

Abstract: The use of copyrighted materials in training language models raises critical legal and ethical questions. This paper presents a framework for and the results of empirically assessing the impact of publisher-controlled copyrighted corpora on the performance of generative large language models (LLMs) for Norwegian. When evaluated on a diverse set of tasks, we found that adding both books and newspapers to the data mixture of LLMs tend to improve their performance, while the addition of fiction works seems to be detrimental. Our experiments could inform the creation of a compensation scheme for authors whose works contribute to AI development.

## introduction generative language models have radically reshaped the landscape of natural language processing (nlp), enabling the development of systems that can generate and interact with human language at an unprecedented level. this includes norwegian, for which several large language models (llms) have been trained and published in the recent years using different architectures and licensing choices (kummervold et al., 2021; kutuzov et al., 2021; samuel et al., 2023, 2025; liu et al., 2024). however, the vast quantities of data required for training these models often include copyrighted materials, presenting novel challenges related to intellectual property rights and compensation. additionally, prior research has highlighted significant concerns about dataset composition and quality in large-scale web-crawled datasets, emphasizing the need for more responsible data curation practices (kreutzer et al., 2022; artetxe et al., 2022; penedo et al., 2024). together, these challenges have led to numerous lawsuits across jurisdictions, fundamentally questioning the legitimacy of training models on copyrighted data without explicit permissions from content creators (panettieri, 2024; madigan, 2024; weisenberger et al., 2024). @xcite the first wave of lawsuits emerged shortly after the public release of advanced generative ai models (see appendix a). content creators, including authors, visual artists, and musicians, began to express concerns about the unauthorized use of their work in training datasets. multiple classaction lawsuits were filed in the united states, accusing prominent ai companies such as ope-nai and meta platforms of infringing on copyright laws by using copyrighted materials without obtaining explicit permissions. the authors argued that the unauthorized use of their works without any form of compensation or recognition undermines their intellectual property rights and jeopardizes their ability to earn a living from their creative endeavors. in europe, a coalition of news publishers has taken legal action against google and meta platforms, arguing that the use of journalistic content in training models without fair re-muneration constitutes a breach of copyright and undermines the sustainability of high-quality journalism. likewise, norwegian rights-holder organizations representing publishing houses across the country, contacted the government in late 2023 expressing their concerns over the use of their material in training generative language models and demanding some sort of compensation were their contents to be used in the training of generative language models. as a result, the ministry of culture and equality (kultur-og likestillingsdepartemente) instructed the national library to create a data-driven report they could use in order to make informed decisions in the elaboration of a compensation scheme for the authors. led by the national library of norway, a consortium was formed together with the university of oslo and the norwegian university of science and technology under the umbrella of the so-called mímir project. @xcite in this context, and under the umbrella of mímir, this paper describes a first attempt at empirically evaluating the impact of copyrighted content in the training of llms for norwegian. we introduce a set of carefully curated datasets that are later used in the training of foundational, domain-tuned, and instruction-tuned models. we establish the proper training conditions to be able to compare models trained on the different datasets. a newly created benchmarking suite is used to evaluate the performance of each individual model and make the comparison meaningful. as a collaborative effort among several institutions, the results of our investigations set the basis to guide policymaking and proper compensation schemes for authors and right-holders in norway (nasjonalbiblioteket, 2024). ## methodology the methodology involves a comprehensive analysis that spans several stages. initially, a diverse corpus of primarily norwegian language data is assembled, incorporating both copyrighted and non-copyrighted materials, plus materials commonly found on the internet. this corpus serves as the foundation for training various llms, each with different configurations and access levels to copyrighted content. by comparing the performance of these models across a range of linguistic and natural language processing tasks, such as text generation, translation, summarization, questionanswering, sentiment analysis and more, we seek to quantify the specific contributions of copyrighted materials to the overall model quality. to ensure robustness and reliability, the evaluation framework focuses on generation capabilities, natural language understanding, and linguistically-inspired metrics. quantitative measures include traditional nlp metrics like accuracy, f1, bleu, and rouge, which provide assessments of model accuracy and fluency. linguistic analysis, on the other hand, involves assessing the coherence, language variability, and contextual relevance of the generated outputs. this dual approach allows for a nuanced understanding of how copyrighted materials impact the performance and utility of llms. ## data collection with the objective of setting up a realistic training scenario where using internet crawled sources is commonplace, we gathered publicly available text collections like wikipedia, datasets from the hplt (de gibert et al., 2024) and culturax (nguyen et al., 2024) projects, code in different programming languages from lozhkov et al. (2024), governmental reports and publications published under open licenses, and books and newspapers articles in the public domain. we then collaborated with the national library of norway and the rights-holder organizations to gain access to protected materials. through the legal deposit act, the national library of norway has digitized almost all books in norwegian and around 85% of the newspapers ever published in the country (nasjonalbiblioteket, 2024). where the quality of the digitized material was not enough (e.g., due to ocr processing), or was not been legally deposited (e.g., paywalled content), specific agreements were put in place to obtain the material from third party organizations such as the norwegian broadcasting corporation (nrk), the tv channel tv2, and the newspaper conglomerates amedia and schibsted. in line with provisions that allow research on language technology and data mining ( åndsverkloven), and with the consent of the norwegian right-holders, this study primarily relied on material legally deposited at, or under agreement with, the national library of norway. specifically, we focus our study on the collection of publisher-controlled books and newspapers ar-dataset documents words base 60,182,586 40,125,975,241 extended 125,285,547 82,149,281,266 ## model training the training phase involved multiple models, each based on the mistral architecture (jiang et al., 1. to measure the overall impact of publishercontrolled copyrighted corpora and its effect in realistic scenarios, we conducted pretraining on the base and extended datasets, both from scratch and using the existing weights (warm) of the pre-trained model mistral 7b v0.1. @xcite these four core models were trained on the same amount of data, 64,000 steps of 4 million sub-word tokens each. using identical sets of hyperparameters (see overall, we trained 17 models (7 billion parameters each) using a total of 270,000 gpu-hours. model training specifications are shown in model initialization gpu/hours accelerator core models base from scratch 50k amd mi250x extended from scratch 50k amd mi250x base (warm) mistral 7b v0.1 13.8k nvidia h100 extended (warm) mistral 7b v0.1 55.6k amd mi250x domain tuned models base + fiction books base 7.5k amd mi250x base + nonfiction books base 7.5k amd mi250x base + nonfiction books + newspapers base 7.5k amd mi250x base + newspapers base 4.8k google tpuv4 base + books base 4.8k google tpuv4 base + books + newspapers base 4.8k google tpuv4 base + original books + newspapers base 9.1k amd mi250x base + original books base 9.1k amd mi250x base + translated books base 9.1k amd mi250x instruction fine tuned models base instruct base 14.2 nvidia h100 extended instruct extended 14.2 nvidia h100 base (warm) instruct base (warm) 14.2 nvidia h100 extended (warm) instruct extended (warm) 14.2 nvidia h100 tpus through the tensor research cloud program foot_6 . besides, we trained two tokenizers with the base and extended datasets separately, both with the same vocabulary size of 32, 768. after an initial test of the fertility of the tokenizers, @xcite we found the difference between them was only 0.0013. therefore, we decided to use the same tokenizer trained with the base dataset for all the models. ## evaluation framework in our empirical evaluation experiments, we utilize noreval, foot_8 a publicly available framework for evaluating norwegian generative llms built on lm-evaluation-harness (gao et al., 2024). we consider 28 tasks, which test model's various norwegian language understanding and generation abilities. noreval covers both norwegian language varieties (bokmål and nynorsk) and provides a set of 4-6 prompts for each downstream task. the tasks can be grouped into nine higher level skills: we follow the standard in-context learning evaluation design for pretrained decoder-only language models (e.g., brown et al., 2020; touvron et al., 2023), which includes zero-shot and fewshot evaluation. in this paper, for the sake of simplicity, we selected the most common metrics per task and aggregated scores using a simple cumulative sum per higher-level skill. in order to aggregate results into an overall score, with the caveats of aggregating metrics of different nature, scores were extracted for the best available {0, 1, 4, 16}shot configuration for each task and the best score for each of the prompts. metrics were normalized to exhibit the same higher-is-better behavior in a range of 0 to 100. ## results the evaluation of the trained models demonstrated that incorporating publisher-controlled copyrightprotected corpora provided a measurable boost in performance across a range of nlp tasks. to illustrate the overall performance differences, figure 2 shows the total scores across all skills, averaged by task for each model. non-aggregated scores mistral 7b v0.1 base (warm) extended (warm) base + translated books base + original books + newspapers base + original books base + nonfiction books + newspapers base + nonfiction books base + fiction books base + books + newspapers base + newspapers base + books base extended 441.83 413.98 427.30 440.35 435.64 408.20 427.40 440.97 427.87 436.81 409.54 482.98 480.28 448.01 sentiment analysis fairness & truthfulness reading comprehension world knowledge commonsense reasoning norwegian language summarization translation variation & readability for all tasks, prompts, and models are available at the mímir repository. 17 ## discussion our findings underline the value of copyrighted materials in improving the performance of generative language models, particularly for specialized nlp tasks in norwegian. the inclusion of these curated publisher-controlled texts provide a substantial advantage in terms of language richness, coherence, and context-specific understanding. however, these advantages are significantly less evident in models that are warm-started using weights pre-trained on other languages, primarily english. we see two possible reasons for this: due to the lack of transparency regarding the exact composition of training datasets in models like mistral, concerns about potential data contamination remain relevant. this overlap could explain why continuous pretraining on similar content did not yield the expected benefits for the warm-started extended models (li et al., 2024; dong et al., 2024; xu et al., 2024; samuel et al., 2024). ## conclusion our study represents a pioneering effort to quantify the impact of copyrighted materials on llms for norwegian. our results indicate that highquality publisher-controlled copyrighted content significantly enhances model performance, especially for complex nlp tasks. however, these benefits bring forth ethical and legal challenges that must be addressed to ensure a sustainable and fair approach to llm development. by providing empirical evidence, we aim to contribute to the ongoing discourse on the role of copyright in ai and inform future policies that support both innovation and the rights of content creators.