Temporal Domain Adaptation for Historical I rish (2023)

Abstract: The digitisation of historical texts has provided new horizons for NLP research, but such data also presents a set of challenges, including scarcity and inconsistency. The lack of editorial standard during digitisation exacerbates these difficulties. This study explores the potential for temporal domain adaptation in Early Modern Irish and pre-reform Modern Irish data. We describe two experiments carried out on the book subcorpus of the Historical Irish Corpus, which includes Early Modern Irish and pre-reform Modern Irish texts from 1581 to 1926. We also propose a simple orthographic normalisation method for historical Irish that reduces the type-token ratio by 21.43% on average in our data. The results demonstrate that the use of out-of-domain data significantly improves a language model’s performance. Providing a model with additional input from another historical stage of the language improves its quality by 12.49% on average on non-normalised texts and by 27.02% on average on normalised (demutated) texts. Most notably, using only out-of-domain data for both pre-training and training stages allowed for up to 86.81% of the baseline model quality on non-normalised texts and up to 95.68% on normalised texts without any target domain data. Additionally, we investigate the effect of temporal distance between the training and test data. The hypothesis that there is a positive correlation between performance and temporal proximity of training and test data has been validated, which manifests best in normalised data. Expanding this approach even further back, to Middle and Old Irish, and testing it on other languages is a further research direction.

## introduction with the increasing digitisation of historical texts, more data becomes available for analysis alongside contemporary documents. however, such data poses a set of challenges for any nlp task as it tends to be both scarce and inconsistent. apart from natural artefacts of language evolution, such as spelling variation and grammatical changes, working with historical languages is complicated by the lack of a linguistic / editorial standard when this data is being digitised @xcite @xcite . it is especially true for early irish, as @xcite and @xcite have pointed out. in this work, we explore the possibility of temporal domain adaptation foot_0 on early modern irish and pre-reform modern irish data. although these are not the oldest stages of the irish language, they are less resourced and more versatile than modern irish, which is itself a minority language. we conduct a set of experiments on the use of out-ofdomain data, both later and earlier than the target time period, for pre-training embedding models to improve the quality of a language model at the said period. we also investigate the effect that temporal distance between embedding training data and test data has in such a setting. finally, we propose a simple and efficient normalisation method for historical irish. ## related work the surge of interest in distributional semantics has lately reached historical linguistics. a recently emerged concept of diachronic, or dynamic @xcite @xcite , embeddings transforms the task of language modelling into the task of modelling language change, which most papers in this field focus on @xcite @xcite @xcite @xcite @xcite . in 2018, three comprehensive surveys of detecting and measuring semantic shifts with word embeddings came out @xcite @xcite . in 2020, one of the semeval shared tasks was dedicated to unsupervised lexical semantic change detection @xcite . at least two phd theses on the topic, "distributional word embeddings in modelling diachronic semantic change" @xcite and "models of diachronic semantic change using word embeddings" @xcite , have been defended in the last few years. less attention has been paid to addressing the challenges historical languages pose for training a robust embedding model, such as high spelling variation or substantial grammatical change over time. a good example of such a work is a paper by @xcite , who discuss the effectiveness of different algorithms for embedding training in diachronic low-resource scenarios and propose improvements to initialisation schemes and loss regularisation to deal with data scarcity. di @xcite are suggesting to use atemporal compass vectors as heuristics while training diachronic word embeddings on scarce data. on the other hand, the use of closely related languages or language varieties to improve word embeddings and language models in a low-resource setting has been a subject of active discussion. for example, @xcite model a low-resource scenario on spanish data, using italian and portuguese as donor languages for training a statistical machine translation model. @xcite work in real low-resource conditions, successfully using turkish data to improve a language model for uyghur. @xcite make another successful attempt at leveraging better-resource turkic languages to improve the quality of the embeddings for related low-resource languages. @xcite achieve a better performance on the low-resource tibetan language by training cross-lingual chinese-tibetan embeddings. generally, transfer learning is a popular approach in neural machine translation when it comes to the lack of data, as described in @xcite ; nguyen and chiang (2017); @xcite ; @xcite ; @xcite . however, the cross-lingual transfer aimed at overcoming data scarcity is not limited to related languages @xcite . the problem of low-resource scenarios is also discussed in an extensive survey of the cross-lingual embedding models @xcite . a few works consider the transfer between different historical stages of the same language as a case of domain adaptation @xcite @xcite , and we adopt this terminology. manjavacas and fonteyn (2022) compare adapting and pre-training large language models for historical english, concluding that pre-training on domainspecific (i.e. historical) data is preferable despite being costly and dependent on the amount of training data. however, the effect on a language model's performance produced by initialising it with temporarily distant pre-trained embeddings and by using the out-of-domain temporal data at the training stage has not been evaluated yet, to the best of our knowledge. moreover, the irish data has never been used in the research on diachronic word embeddings and temporal domain adaptation before. ## data the data for the experiment is a collection of early modern irish and modern irish texts spanning over 350 years, from the late 16 th to early 20 th century. irish belongs to the celtic branch of the indo-european language family. like other celtic languages, it is notable for initial mutations: sound changes at the beginning of a word happening in certain grammatical environments, which are reflected in spelling. these are combined with a rich nominal and verbal inflection at the end of a word. the four types of initial mutations in modern irish and their effect on spelling is shown in before becoming a grammatical feature of the language, mutations happened as historical phonetic processes. foot_1 for instance, a mutation called lenition in the intervocalic position turned old irish cride ["k j r j ið j e] 'heart' into middle irish croid(h)e / cridhe / craid(h)e ["k (j) r (j) ig j @] / ["k (j) ## conclusion the results cited above testify that using out-ofdomain temporal data in the pre-training and training of a language model for a historical language can significantly improve its performance. this is extremely valuable in low-resource scenarios, where we may only have a few texts dating back to a particular period, which would not be enough to train a robust language model. providing a model with additional input improves its quality by 12.49% on average on non-normalised texts and by 27.02% on average on normalised texts even if this information is retrieved from data covering a different -no matter later or earlier -period in the history of a language. most importantly, using only out-of-domain data at both pre-training and training stages allows for achieving up to 86.81% of the baseline model quality on non-normalised texts and up to 95.68% on normalised texts without any target domain data. our hypothesis that there is a positive correlation between the performance of language models and the temporal proximity of training and test data has been validated. this effect manifests best in orthographically normalised texts. expanding this approach even further back, to middle and old irish, and testing it on other languages is a further research direction. finally, we proposed a simple yet very effective orthographic normalisation method for historical irish that reduced the type-token ratio by 21.43% on average in our data and allowed for up to 33.02% drop in a language model's perplexity.