HPLT : High Performance Language Technologies (2023)

Abstract: We describe the High Performance Language Technologies project (HPLT), a 3-year EU-funded project started in September 2022. HPLT will build a space combining petabytes of natural language data with large-scale model training. It will derive monolingual and bilingual datasets from the Internet Archive and CommonCrawl and build efficient and solid machine translation (MT) as well as large language models (LLMs). HPLT aims at providing free, sustainable and reusable datasets, models and workflows at scale using high-performance computing (HPC).

## introduction the hplt project aims at innovating the current language and translation modelling landscape by building the largest collection of free and reproducible models and datasets for around 100 languages. datasets will be derived from webcrawled data using already established processing pipelines from the paracrawl 1 and macocu corpora. 2 they will be adapted and improved to run efficiently on hpc centres in order to produce consistent datasets at scale. hplt will also build open, sustainable and efficient llms and mt models with significant language coverage using the powerful supercomputing infrastructure of european hpc centres. datasets, models, pipelines and software to build them will be shared along with additional tools to ease data management, model building and evaluation. an hpc-powered consortium: the consortium gathers research groups, the experience of an in-dustry partner, and the computational infrastructure and involvement of two hpc centres in europe. most of the processing will happen on lumi, a pre-exascale supercomputer, which will be made nlp-aware to pave the way for further initiatives and exploitation of the project outcomes. the 8 partners in the consortium are: charles university in prague, university of edinburgh, university of helsinki, university of oslo, university of turku, prompsit language engineering, cesnet, and sigma2 hpc centres. ## expected results datasets: starting from 7 pb of web-crawled data from the internet archive foot_1 and 5 from com-moncrawl, foot_2 we will derive monolingual and bilingual datasets for systematic llm and mt building with a large language coverage. data curation, a crucial part of the process, will be based on adapted versions of the bitextor and monotextor pipelines foot_3 . filtered and anonymized versions enriched with genre information will be released. output formats will follow commonly adopted standards and their distribution will be handled through opus foot_4 and lindat foot_5 with opensource licenses along with analytics and metadata. models: efficient and high-quality language and translation models will be built and released. regarding llms, when sizes and computational resources allow, we aim at building bert @xcite , t5 @xcite , and gpt-like models @xcite ) for all the targeted languages. we will opt for multilingual models where necessary to mitigate the lack of sufficient training data that is expected for some of the targeted languages. for mt models, we plan to build not only english-centric models but also other language combinations including multilingual models depending on data availability and interest. we will share hplt models through opus-mt and huggingface with open-source licenses. the first hplt llms have already been published: gpt3like models for finnish 8 , still under evaluation. pipelines and tools: hplt wants to ease data management and model building, making hpc centres in europe ready to run the same pipelines and tools in a transparent and straightforward manner even on other datasets and languages. below, we describe two of the tools that hplt is developing in this direction. opuscleaner 9 is a one-stop dataset download/examine/filter toolkit built with modern largescale nlp models in mind. it is based on python and uses a web interface to make it easy to run on hpc clusters. the workflow is as follows: (1) dataset selection: downloads to the host running the web server, not the local machine; (2) filter selection: allows filtering and visualizing the effect interactively on a random sample of each selected dataset; (3) labeling: allows categorising each dataset; (4) batch filter execution: applies filters and labeling to all datasets from a one-line runme command and (5) dataset (near-)deduplication across collections. opustrainer 10 is a large-scale data shuffler/augmenter which takes a collection of datasets and feeds it to a neural network training toolkit according to a set schedule. its design aims to solve neural network training problems at scale. it features: (1) sampling and mixing of data from multiple sources; (2) per-source shuffling and independent dataset mixing avoiding out-of-memory issues; (3) curriculum learning with the definition of training stages, each one having its own mixture of datasets; (4) stochastic modifications of the training batch to support end-user requirements like support for title case, all caps, placeholders, etc. ## mt at hplt hplt's ambition is to democratise access to efficient mt. we will use our large curated datasets with robust software pipelines to train high-quality mt systems and, by leveraging the hpc capacity available to the project, over an extensive set of 8 https://turkunlp.org/gpt3-finnish 9 shorturl.at/bolw7 10 shorturl.at/pdkpt languages. all models will be properly evaluated and documented using standard metrics. releasing all models with appropriate metadata and optimised training recipes will also help to avoid unnecessary computation for sub-optimal and repetitive procedures. beyond large systems, we aim to build lightweight models using knowledge distillation @xcite ). an ensemble of large teacher models can produce compact students that mimic their teacher's quality, with negligible degradation but much lower computational costs during inference. quantisation and other efficiency techniques can further increase speed and lower the memory footprint, which is essential for responsive and large-scale translation tasks.