Do not Trust the Experts - How the Lack of Standard Complicates NLP for Historical I rish (2023)

Abstract: In this paper, we describe how we unearthed some fundamental problems while building an analogy dataset modelled on BATS (Gladkova et al., 2016) to evaluate historical Irish embeddings on their ability to detect orthographic, morphological and semantic similarity.performance of our models in the analogy task was extremely poor regardless of the architecture, hyperparameters and evaluation metrics, while the qualitative evaluation revealed positive tendencies. argue that low agreement between field experts on fundamental lexical and orthographic issues, and the lack of a unified editorial standard in available resources make it impossible to build reliable evaluation datasets for computational models and obtain interpretable results. We emphasise the need for such a standard, particularly for NLP applications, and prompt Celticists and historical linguists to engage in further discussion. We would also like to draw NLP scholars’ attention to the role of data and its (extra)linguistic properties in testing new models, technologies and evaluation scenarios.

## introduction historical languages are known to present greater challenges to nlp due to high orthographic variation, diachronic morphological changes and lack of resources @xcite @xcite . our initial goal was to compare different embedding architectures and hyperparameters for detecting morphological and spelling variation in historical irish, but we unearthed some fundamental problems while we were building an evaluation dataset and testing our models on it. ## word embedding evaluation scenarios there are two main strategies for the evaluation of word embeddings: extrinsic and intrinsic @xcite @xcite . extrinsic evaluation involves using pretrained embeddings as input vectors in a model solving a downstream nlp task, such as part-ofspeech tagging, named entity recognition, or sentiment analysis. the model's performance is believed to reflect the quality of the embeddings it was initialised with. intrinsic evaluation is focused on assessing linguistic relations within the embedding model itself through solving specially designed mathematical problems: similarity and analogy. the similarity task entails comparing the similarity scores of two words yielded by an embedding model to those calculated based on experts' judgments. the analogy task is a vector proportion, where we ask an embedding model, "what is to b as a ′ is to a?", and expect b ′ as an answer. generally, task-driven extrinsic evaluation looks more feasible, because it allows the use of already existing evaluation datasets. however, the majority of downstream tasks have not been attempted yet for many minority and historical languages, which leaves us with no available datasets or baselines. as such, constructing a small dataset for intrinsic evaluation seems the best alternative. both analogy and similarity datasets can be created automatically or semi-automatically by translating an existing dataset from another language, or with the help of a wordnet or a comprehensive dictionary of a language in question in a machine-readable format if there are any. such a dataset would still require expert proofreading and evaluation, but the amount of manual work would not be as daunting as when a dataset is created from scratch. ## early irish analogy dataset traditionally, analogy datasets are based on pairwise semantic proportion @xcite , and therefore every question has a single correct answer. given the high level of variation in historical languages, such a strict definition of a correct answer seems unjustified. therefore, we follow the creators of the bigger analogy test set, or bats @xcite . this dataset has highlighted the problems of popular embedding models, such as glove, and provided additional proof of the importance of subword information for capturing morphological relations. the origi-nal english bats has successfully been adapted to japanese @xcite and @xcite . our early irish analogy dataset is not a full-scale adaptation of bats but draws heavily upon the ideas behind it, providing several correct answers for each analogy question and evaluating the performance with set-based metrics proposed by bats authors, such as an average of vector offset over multiple pairs (3cosavg) and a logistic regression cosine similarity (lrcos): the early irish analogy dataset consists of four parts: morphological variation, spelling variation, synonyms, and antonyms. the morphological and spelling variation data was automatically extracted from the edil @xcite , a digital historical dictionary of medieval irish covering the period ca. 700 -1700. spelling variants were taken from the headwords, and the morphological variation subset was compiled from the 'forms' field that covers both inflected forms of a headword and its derivatives. unlike the original bats, no division was made between different types of inflection, nor between inflection and derivation, within the morphological variation subset because the structure of edil does not allow for obtaining this division automatically. we would also like to point out that the edil sometimes lists spelling variants along with inflected forms and derivatives in the 'forms' section, and we did not filter them out manually. the raw data amounted to 2,370 spelling variation and 9,690 morphological variation questions, from which 100 examples were randomly selected for each of the subsets to be comparable in size with the synonym and antonym subsets. the synonym and antonym subsets are translations of the correspondent bats parts proofread by four expert evaluators. the translations for each entry in the synonym subsets l07 (intensity, cry : scream) and l08 (exact, sofa : couch), and antonym subsets l09 (gradable, clean : dirty) and l10 (binary, up : down) were obtained by reversesearching the edil. the translations were then organised in synsets, each labelled with an english keyword, which the expert evaluators were asked to review. the evaluators were allowed to consult edil but were advised not to rely on provided definitions, if in doubt, but instead to utilise their knowledge of how these words occur in texts. the task description also included the following guidelines: we obtained 98 entries in the synonym subset and 109 entries in the antonym subset, upon which three or more experts agreed. if a word had multiple spellings in the corresponding edil entry, we included all of them in these subsets. ## experiment, evaluation and epic fail our initial goal was to compare different embedding architectures to measure the effect of leveraging subword information on detecting morphological and spelling variation along with semantic similarity in a diachronic scenario. we also aimed at finding the best embedding size for our lowresource and highly inconsistent data. for this purpose, we trained skipgram @xcite and fasttext @xcite models with embedding sizes of 20, 50, 100 and 300 on old and middle irish corpora, as well as on both of them combined. we refer to the combined old and middle irish data as 'early irish' for convenience, although this term is usually used to describe a broader period, from primitive irish (4 th -6 th c. a. d.) to middle irish (10 th -12 th c. a. d.), according to @xcite . more information about the training data for embedding models is provided in first, we made a few queries to the biggest early irish fasttext model foot_1 to see the word vectors nearest to these queries. for example, the closest words to mainister 'monastery' were its spelling variants (mainistear, mainistir, mainisttir), forms with suffixed demonstratives (mainistir-si, mainistir-se, mainisttir-si, mainistir-sin) and compounds (cédmhainistir 'early monastery, former monastery', énmhainistir 'individual monastery'). the name of a legendary irish king, ailill, yielded spelling variants (ailil, oilill), mutated and inflected forms (hailill, tailill, aililla) 3 and another personal name, ailill miltenga. the early irish skipgram model with the same parameters did not capture any morphological or spelling variation but detected semantic associates for personal names from the early irish literature. then, we used the tensorflow projector @xcite to see if there are any meaningful clusters in the 3d projection of the vector space of the aforementioned early irish fasttext model. we found many interesting clusters of different sizes, such as nouns referring to peoples perceived as foreign in the dat. pl. (allmurachaib 'to foreigners', lochlannachaib 'to scandinavians', saxanachaib 'to saxons', paganachaib 'to pagans') or verbal nouns ending in -udh (etargnaghudh 'interpreting, explaining', cotludh 'sleeping', slonnudh 'naming, mentioning' etc.). it is worth mentioning that the model learned subtle spelling differences: the first cluster mentioned above did not include the later spelling variants with the ending -aibh, and in the same way, the second cluster did not include earlier spelling variants ending in -ud rather than -udh. moreover, nouns with a suffixed demonstrative sin formed two different clusters depending on whether the demonstrative was hyphenated (fechta-sin, sliabh-sin, caislein-sin etc.) or not (ceilgsin, uairsin, curuchsin etc.). thus, we witnessed that our models did learn a significant amount of spelling variation, as well as some inflectional and derivational morphology patterns and a limited quantity of semantic similarities. in this case, what factors may have contributed to the inadequate performance observed? ## conclusion we discussed an attempt at building an analogy dataset to evaluate historical irish embeddings on their ability to learn orthographic, morphological and semantic similarity. however, the performance of our models was extremely poor regardless of the architecture, hyperparameters and evaluation metrics, while the qualitative evaluation revealed positive tendencies. several factors have contributed to it, including a low agreement between experts on fundamental lexical and orthographic issues, and the lack of a unified editorial standard for the language. these problems are by no means caused by poor scholarly practice. each of the electronic resource creators pursues a particular, perfectly justifiable editorial approach that dictates their choices. however, the necessity of a text editing standard, especially for nlp applications, has not been properly debated and investigated by the historical irish academic community. we suspect that this may be the problem of historical languages in general. through this paper, we would like to highlight this issue and invite celticists and historical linguists to engage in further discussion.