HPLT ’s First Release of Data and Models (2024)

Abstract: The High Performance Language Technologies (HPLT) project is a 3-year EU-funded project that started in September 2022. It aims to deliver free, sustainable, and reusable datasets, models, and workflows at scale using high-performance computing. We describe the first results of the project. The data release includes monolingual data in 75 languages at 5.6T tokens and parallel data in 18 language pairs at 96M pairs, derived from 1.8 petabytes of web crawls. Building upon automated and transparent pipelines, the first machine translation (MT) models as well as large language models (LLMs) have been trained and released. Multiple data processing tools and pipelines have also been made public.

## introduction the hplt project combines petabytes of natural language data and large-scale model training. focusing on different aspects of the project, there are eight partners in the consortium: charles university in prague (coordinator), university of edinburgh, university of helsinki, university of oslo, university of turku, prompsit language engineering, and cesnet and sigma2 hpc centres. the project has achieved several milestones in its first half @xcite . specifically, 1) we developed essential data tools and training pipelines; 2) we successfully produced the first edition of datasets for 75 languages and 18 language pairs; 3) we trained and released the first batch of llms and mt models. ## future plan for machine translation in our next mt model release, we target three aspects. first, we aim to deliver mt models for even lower-resourced languages that hplt has acquired. next, we will investigate cost-efficient mt models via distillation, quantization, etc. finally, in comparison with the current release of unidirectional translation models, we plan to explore massively multilingual models and large language models.