e STÓR : Curating I rish Datasets for Machine Translation (2025)

Abstract: Minority languages such as Irish are massively under-resourced, particularly in terms of high-quality domain-relevant data, limiting the capabilities of machine translation (MT) engines, even those integrating large language models (LLMs). The eSTÓR project, described in this paper, focuses on the collection and curation of high-quality Irish text data for diverse domains.

## introduction despite the growing ubiquity of digital technologies, the irish language lacks robust language technology that serves irish speakers adequately in the digital sphere, with irish language classified as in the "weak or no support" category of european languages @xcite . this digital disconnect poses a significant threat to the vitality and sustainability of the irish language resulting in the very real threat of digital extinction in the medium to long term. the digital plan for irish @xcite @xcite is a detailed guide regarding areas in irish language technology that require development. the estór (sonraí teanga óstáilte i gcomhair ríomhphróiseála "hosted language data for digital processing") project is funded by the irish government (department of tourism, culture, arts, gaeltacht, sport and media) from 2021 to 2025, to address the lack of high-quality data described in the digital plan, by providing a digital platform ( https://estor.ie/ ) for sharing bilingual and monolingual irish text data. in addition to data collection, the project aims to further research and technological innovation, promote language accessibility, and educate members of the public and government bodies on the value of irish language data. additionally, the project collaborates with the european commission to © 2025 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. ## data curation and back-translation experiments text data shared to the estór platform originates mainly from individuals or organisations in the public or government bodies of ireland, often through direct contact. to date, 188 parallel language resources, totalling 185,343 translation units have been uploaded and processed on the estór platform, and 201,719 words of monolingual data. while sourcing the data from trusted language producers encourages reuse of existing high-quality data sources, and spreads awareness of the importance of data sharing, this resource-intensive approach is difficult to scale in order to meet the increasing demands for large data collections. additionally, the text types collected from these sources offer limited variety of style, tone, and topic, resulting in unbalanced coverage in nlp models. to address these issues, the estór project has begun experimenting with alternative methods of data collection and production. web crawling is a popular method of sourcing large quantities of language data that can be largely automated, but the quality is difficult to ensure. employing a blend of manual inspection and automatic filtering, the estór project has experimented with selecting high-quality articles from irish wikipedia vicipéid ( https://ga.wikipedia.org/ ), and using the etranslation irish-to-english general model to perform back-translation to generate synthetic parallel datasets covering diverse topics. this dataset can then be employed as test data to investigate coverage of existing irish mt models. ## data cleaning much of the importance of the estór project lies in the data cleaning work, which requires meticulous attention to detail and a robust knowledge of irish and english language to ensure correctlyaligned, relevant, and high-quality clean data. the impact of this essential work in the development of powerful nlp tools and applications is still underplayed in the larger nlp community @xcite , and precise details of the task can often be glossed over or omitted in reporting. there are many components to process the data in its raw form, although at a minimum the following steps are undertaken: initial input assessment: typically files uploaded to estór are aligned (e.g. translation memories, spreadsheets, aligned plain-text), unaligned editable (e.g. word processing types, unaligned plain-text), or unaligned uneditable (e.g. pdf generated by software). raster formats are typically not accepted due to the additional challenge of running optical character recognition, although scans of hard-copy data are currently being processed as part of collaborative digitisation work. text extraction and normalisation: while text file types are processed using hand-written text extraction tools, library support is used for extracting text from binary file types (e.g. pdfs). normalisation of text encoding maintains consistency throughout the data, and helps prevent incorrectly encoded characters or unicode-equivalence errors. language identification: text is sampled at regular intervals throughout the dataset to perform language identification and ensure that the file contains the correct language. a standardised irish language model was trained for this task, using the langdetect 1 python port of the original tool @xcite . sentence splitting: it is often necessary to reconstruct sentence boundary information in order to produce the sentence-aligned output. this task can be as trivial as splitting on sentence-final punctuation (e.g. '.', '?'), but becomes more challenging when processing text containing e.g. abbreviations (e.g. 'etc.', 'dr.'). as abbreviations in irish differ from english (e.g. uimhir 'number' is abbreviated as 'uimh.'), it was necessary to define bespoke rules to process most of these cases automatically. document and sentence alignment: while unmatched irish text files can be published as mono-1 https://pypi.org/project/langdetect/ lingual data, any files uploaded in english must be aligned with an irish file to be considered of use. sentence alignment ensures that the text on each line of each aligned file pair corresponds with the text on that same line in the other language, employing the hunalign @xcite tool. verification: the final step is to assess aligned document pairs to ensure that the data has been correctly processed according to specified criteria (e.g. numerals appearing on one side should appear on the other). the text is checked through a series of automatic checks, and potential bad alignments are flagged for manual review. ## conclusion we present the estór project, an effort in curating and cleaning high-quality irish text data for the development of language technology, including improved mt engines. the project has many components, but this paper focuses on the data cleaning and selection tasks, which constitutes a vital step in the development of any nlp applications.