HPLT ’s Second Data Release (2025)

Abstract: We describe the progress of the High Performance Language Technologies (HPLT) project, a 3-year EU-funded project that started in September 2022. We focus on the up-to-date results on the release of free text datasets derived from web crawls, one of the central objectives of the project. The second release used a revised processing pipeline, and an enlarged set of input crawls. From 4.5 petabytes of web crawls we extracted 7.6T tokens of monolingual text in 193 languages, plus 380 million parallel sentences in 51 language pairs. We also release MultiHPLT, a cross-combination of the parallel data, which produces 1,275 pairs, as well as releasing the containing documents for all parallel sentences in order to enable research in document-level MT. We report changes in the pipeline, analysis and evaluation results for the second parallel data release based on machine translation systems. All datasets are released under a permissive CC0 licence.

## introduction the hplt project runs from 2022 to 2025, and focuses on the processing petabytes of natural language data and large-scale model training. the consortium is made of eight partners: charles university in prague (coordinator), university of edinburgh, university of helsinki, university of oslo, university of turku, prompsit language engineering, and cesnet and sigma2 hpc centres. following the previous release at the end of 2023 (de gibert et al., 2024), the project has recently completed the release of a new massive multilingual dataset for both monolingual and parallel data along with improved pipelines and tools extensively described in (burchell et al., 2025). parallel data stats and analysis the second release of the hplt parallel data covering 51 language pairs contains 380,710,720 sentence pairs with 6,779,910,082 english words. our selection of pairs avoided the top 20 highest resourced (according to opus) and focused on the next ranked languages, in order to maximise impact. the sizes of the different language pairs show significant variation, with a median of 3,927,371 sentence pairs. this range spans from 273,430 sentence pairs for english-sinhala to 29,067,875 sentence pairs for english-finnish, the largest parallel data set. we get a 36% increase in the number of sentences compared to the first release for the 18 overlapping languages. during filtering, 40% of the parallel sentence pairs are eliminated and an additional 50% is removed due to deduplication. the final corpus shows a 70% decrease in sentence pairs relative to the raw data. this reduction is less significant than in the first release, which we assume is due to starting with cleaner monolingual text. we inspect the data with the hplt analytics tool. @xcite we find that small-sized datasets contain larger portions of wikipedia and religious content while medium/large-sized ones contain high-portions of hotel booking and travel websites. some popular domains include websites from gaming, software or e-commerce translated into a big number of languages, probably using mt. from the inspection of the most frequent n-grams, we find that they are very similar across all parallel datasets, especially among larger ones, frequently related to hotels and legal notices. extrinsic evaluation of the parallel data we train bidirectional mt models on the new released parallel data to extrinsically evaluate the performance of the released datasets. we compare models trained on only hplt data for both releases and, additionally, models trained with hplt data in combination with tatoeba. @xcite we build and release mariannmt compatible mt models for all bitexts in hplt v2 using the same tooling as the one used in the previous release: opuscleaner (data selection and cleaning), opus-trainer (data scheduling and augmenting), and opuspocus (training process management). these tools are fully described in the public deliverable of the hplt project focused on pipelines and tools. @xcite automatic metrics are computed on flores-200 foot_10 for evaluation. results computed on 10 out of the overlapping 18 language pairs between the first and the second release show gains in bleu in favour of hplt v2 mt models going into english with an average gain of 4.2 bleu. from english, the average gain is 3.5 in bleu, with 7 out of the 10 models being better with hplt v2 data and the remaining 3 being on par between the first and second release. when combining hplt v2 data and tatoeba, mt models result in a 7% relative increase in bleu for both translation directions.