From Words to Action: A National Initiative to Overcome Data Scarcity for the S lovene LLM (2025)

Abstract: Large Language Models (LLMs) have demonstrated significant potential in natural language processing, but they depend on vast, diverse datasets, creating challenges for languages with limited resources. The paper presents a national initiative that addresses these challenges for Slovene. We outline strategies for large-scale text collection, including the creation of an online platform to engage the broader public in contributing texts and a communication campaign promoting openly accessible and transparently developed LLMs.

## introduction extremely large language models, such as gpt-4, have demonstrated remarkable advancements across various natural language processing tasks, sparking widespread interest in their applications. however, their reliance on vast and diverse datasets makes them inherently biased toward well-resourced languages. for languages like slovene, with a smaller speaker base and limited data availability, this disparity poses a significant challenge, hindering the development of robust language-specific llms. recent studies have highlighted similar challenges faced by other low-resource languages, underscoring the need for comprehensive multilingual evaluation and language-specific model development. @xcite provide an in-depth assessment of chatgpt's performance across multiple languages, revealing its uneven effectiveness in low-resource contexts. their evaluation, covering seven tasks and 37 languages, exposes significant performance gaps in both lowand extremely low-resource languages. likewise, @xcite explore the broader landscape of llms for low-resource languages, addressing their multilingual, multimodal, and dialectal complexities. their findings emphasize the necessity of language-specific initiatives and reveal the persistent limitations of llms for medium-to lowresource languages, largely due to the lack of representative datasets. while these studies highlight the performancerelated limitations of current llms for lowresource languages, an equally pressing concern is their accessibility. the proprietary nature and high computational demands of existing models restrict access for many research organizations and smaller companies, underscoring the need for open-access alternatives. addressing this issue requires not only the development of computationally efficient models, but also the creation of large, diverse, and high-quality datasets tailored to specific languages. for smaller language communities, such as slovene, the search for texts to be included in llms must extend well beyond readily available online sources, as these alone are insufficient to support the development. in this paper, we introduce a national initiative aimed at overcoming these obstacles for slovene, detailing our strategies for large-scale text collection and community engagement for the development of openly available slovene llms. ## project framework and previous work llms have introduced a major shift in the field of natural language processing (nlp), offering more efficient fine-tuning for common nlp tasks while simplifying their implementation @xcite . the datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of llms @xcite . therefore, preparing language data for llms is a crucial step that directly impacts their performance across various tasks. the effectiveness of nlp tasks relies heavily on the scale of the language model's training, which is directly influenced by access to large, diverse datasets spanning various domains @xcite . moreover, diversity in training data plays a crucial role in enhancing the generalization capabilities of large models, enabling downstream tasks, as outlined in @xcite , to effectively leverage knowledge even with limited training data @xcite . in the ongoing povejmo-adaptive natural language processing with large language models project, foot_0 we aim to develop several computationally efficient and open-access large language models trained on slovene language data. these models will also be adapted and evaluated for selected industry use cases, such as enhancing slovene speech recognition and synthesis for industrial systems, preparing museum materials and interactive systems, as well as for medical applications and infrastructure code generation. we have first gathered 9.2 billion tokens from freely available sources, including existing language corpora and other openly accessible slovene-language data, providing a solid foundation for the project. the initial dataset includes different types of text, such as news articles up to and including september 2023 @xcite , academic works @xcite , web crawls (mc4 @xcite , macocu @xcite , cc100 @xcite ), and various slovene reference and specialized corpora included in the metafida database @xcite ). the gams-1b-chat language model, with one billion parameters, has been trained on this language material @xcite . while this initial model provides valuable insights into the effects of training on slovene data, it is roughly a thousand times smaller than the largest models (e.g., used for the latest version of chatgpt), makes errors frequently, and highlights the need for further text collection to improve performance. 2 3 national text-collection campaign we have estimated that the development of a sufficiently large model requires approximately 40 billion additional words. the estimation of the required amount of training data is heuristic and approximate, derived from two approaches. the first approach is based on the findings of the llama model study @xcite . the study illustrates the decline in the loss function as the number of training tokens increases, where, on average, two tokens correspond to one word. for most models, including llama 7b, which is the closest equivalent to gams, the most significant decrease in the loss function occurs up to approximately 100 billion tokens (that is, around 50 billion words). we expect to collect around 10 billion words from freely available online resources (currently 9.2 billion; see section 2), while an additional 40 billion will need to be gathered using alternative approaches. 3 the second approach relies on llm scaling laws, which suggest that a model of size x requires at least 5x to 10x words for effective training, though in practice the requirement is often even greater. given that the larger gams model is expected to have approximately 10 billion parameters, it would require a minimum of 50 billion words to achieve optimal performance. this ambitious undertaking necessitated the development of a comprehensive communication and operational strategy. the key components of this strategy include: (1) identifying and engaging potential contributors of textual materials; (2) developing efficient mechanisms for text submission; (3) implementing secure and scalable storage systems; (4) establishing effective and reliable processes for tracking and documenting all text acquisition activities; and (5) defining the metadata framework to ensure systematic organization and accessibility of collected texts. beyond these operational aspects, strategy (6) addresses legal and ethical considerations associated with data collection and usage, while also prioritizing (7) promotional and dissemination activities to build public awareness and support. a key objective of these efforts is to emphasize the importance of developing a large-scale language model for slovene, highlighting its far-reaching implications for tech-nological innovation and cultural preservation. our text collection campaign operates through two main strategies. on the one hand, we engage with large-scale text providers such as national libraries, publishing houses, media organizations, government ministries, and other significant contributors, strongly advocating for the value and potential impact of creating a slovene llm and encouraging them to contribute their textual resources. on the other hand, we reach out to individuals, inviting them to donate their own texts to both actively support and directly contribute to the co-creation of the slovene language model. each of these two groups presents unique requirements and demands distinct approaches to communication and engagement. beyond the technical challenges of acquiring and processing the material, the primary obstacles lie in legal constraints. current slovene legislation permits the use of copyrighted material for data mining. however, this does not apply to material available on the web unless it is provided under an appropriate license. however, this does not include the material available on the web. this legal framework imposes significant limitations on the ability to gather and utilize existing slovene texts for model training. addressing this issue involves two potential approaches. the first option is to advocate for legislative reform, urging lawmakers to amend the copyright laws to accommodate the specific needs of language technology development. such a legal adjustment could facilitate broader access to textual resources while ensuring that intellectual property rights are respected in a manner compatible with technological advancements. however, this approach is inherently time-intensive and comes with no guarantee of success. it places the outcome largely outside of our control, as it depends on the willingness of policymakers to adopt the proposed changes and the eventual implementation of new legal frameworks. given these uncertainties, we have determined that a more pragmatic and immediate approach is to actively seek permission from copyright holders to use their texts for the purpose of building a slovene llm. this strategy, while more labourintensive, allows us to make direct progress without waiting for external factors to align. this approach requires substantial effort on our part, as it involves identifying relevant stakeholders, initi-ating discussions, addressing potential concerns, and negotiating agreements. a significant challenge in this process arises from the hesitation of some stakeholders, particularly media outlets and publishers, who are concerned about the uncertain societal implications of artificial intelligence and its potential impact on their work. ## data storage and processing protocol we have implemented a comprehensive protocol for securely managing the storage, encryption, transmission, and processing of data throughout all stages of its lifecycle. data is received through network connections or on portable storage devices, which may contain unencrypted or optionally encrypted data using a temporary key set by the data owner. once received, data undergoes decryption (if necessary) on a secure system, is re-encrypted with a new encryption key, and then securely uploaded to the primary storage system. backup copies of the encrypted data are created and stored on a secondary system to ensure disaster resilience. during processing, unencrypted data is used exclusively for the duration of the analysis or transformation. owners define specific rules for further sharing of their data, such as limiting its use to the project or permitting broader access to other researchers within slovenia, the eu, or beyond. data is transferred between systems using encrypted channels (e.g., ssl/tls, ssh). data minimization principles are applied, providing only the necessary subsets of data for specific use cases, packaged as encrypted "data bundles". all data transfers are tracked to ensure accountability, with detailed records of which data was shared and with whom. processing on high-performance computing systems (e.g., vega supercomputer) or specialized research infrastructures (e.g., frida at the faculty of computer and information science) adheres to strict security protocols. data is stored in encrypted form and decrypted only temporarily during processing. after processing, unencrypted data is securely deleted. future improvements may include leveraging confidential computing technologies for enhanced security. access management prioritizes security and simplicity, given the initial small group of users. encryption keys are manually tracked and managed by the project lead, with backups maintained securely. data bundles for specific use cases are encrypted with unique keys assigned to each user, employing aes symmetric encryption for efficiency and security. as the number of users or datasets grows, more advanced access management systems will be introduced. ## conclusion and future work this paper has outlined the national initiative to overcome data scarcity and support the development of a slovene large language model. by implementing diverse text-gathering scenarios, including a user participation portal and an extensive communication strategy, the project has ac-tively addressed the unique challenges faced by languages with a small speaker base. at the time of this paper's submission, the foundational infrastructure for collecting slovenelanguage texts has been fully established, including a user-friendly portal for contributions and robust protocols for data storage and processing. a nationwide promotional campaign was launched, aiming to mobilize broad participation from both institutions and individuals. by the time of the workshop, we anticipate being able to report on the first campaign's outcomes, including the volume and diversity of collected texts and their suitability for training the slovene large language model. these results will provide valuable insights into the scalability and replicability of such initiatives for other less-resourced languages.