Evaluation of Distributional Semantic Models of A ncient G reek: Preliminary Results and a Road Map for Future Work (2023)

Abstract: We evaluate four count-based and predictive distributional semantic models of Ancient Greek against AGREE, a composite benchmark of human judgements, to assess their ability to retrieve semantic relatedness. On the basis of the observations deriving from the analysis of the results, we design a procedure for a larger-scale intrinsic evaluation of count-based and predictive language models, including syntactic embeddings. We also propose possible ways of exploiting the different layers of the whole AGREE benchmark (including both human- and machine-generated data) and different evaluation metrics.

## introduction the application of natural language processing to the study of ancient greek semantics is an emerging research area which has proven to be a fruitful avenue for our understanding of the ancient greek language and culture. previous work has focused on the training of distributional semantic models (dsms) on ancient greek corpora @xcite @xcite @xcite , a task enabled by the relatively large quantity of extant texts available for this language. dsm evaluation is a necessary step to properly assess the usefulness of applying these models to large-scale studies of ancient greek, but is made particularly challenging by the lack of native speakers and, compared to modern languages, a limited number of experts available. this paper offers an evaluation of dsms for ancient greek against the newly created agree benchmark @xcite and a road map for further, wider evaluation. we exploit the layered nature of agree to assess at different levels four dsms, and discuss results not only in terms of model comparison, but mostly in terms of best evaluation strategies, suggesting various precision-and recall-based options. on that basis, in section 6 we propose a road map for a more comprehensive evaluation campaign, which would involve training a wider range of models, including dependency-based embeddings (see, among others, @xcite @xcite , already preliminarily tested in @xcite , and studying their behaviour with respect to a number of metrics. specifically, we propose to assess the difference in performance between syntactic embeddings trained on manually tagged and on automatically tagged treebanks. we plan to evaluate the dsms, trained with different parameters, against the full version of agree, including both humanand machine-generated judgements. we also suggest alternative ways to use the data collected for agree and possible evaluation metrics. ## previous work few resources exist as gold standards for the evaluation of dsms on ancient greek. @xcite contains the manual annotation of the senses of the lemmas μῦς, ἁρμονία, and κόσμος @xcite and was used in @xcite and @xcite to evaluate models for semantic change detection. @xcite evaluated count-based dsms for ancient greek against benchmarks obtained from an ancient lexicon, a modern dictionary of synonyms, and the computational lexicon ancient greek wordnet @xcite . the data they released represent the first benchmark for the evaluation of ancient greek dsms. 1 reusing preexisting resources, as they did, allows incorporating in the evaluation the semantic knowledge of real speakers of ancient greek (as in the case of the ancient lexicon) and to leverage the semantic knowledge of highly specialized experts, from resources that can be the product of years of work. this data collection seems less biased by the aims of the research, however it also has downsides. lexical resources, compiled by humans, can suffer from idiosyncrasies, for example being biased by the interests and language taste of their author, and if the author is not alive anymore, it is not possible to get explanations about specific choices. moreover, ancient resources can reflect ideas of semantic relationships between words (e.g. word similarity) that are different from the contemporary conceptualization, as also noticed by @xcite and discussed in @xcite . ## training data for dsms of ancient greek the largest corpus of ancient greek, the thesaurus linguae graecae @xcite , containing more than 110 million tokens, 2 is only accessible through the web interface. however, scholars can use a number of open-access machine-readable ancient greek corpora, containing different ranges of text types. 3 some corpora are annotated, for example with lemma, pos, and syntactic information. the diorisis ancient greek corpus (vatri and mcgillivray, 2018), a portion of which was used as training data for the study presented in this paper, contains 10,206,421 automatically lemmatized and postagged tokens. but many corpora with syntactic annotation also exist: an overview of the most often used treebanks for ancient greek is in as the case of glaux shows (see ## the agree benchmark the agree benchmark contains pairs of lemmas semantically related to 36 selected 'seed' lemmas (12 nouns, 12 adjectives, and 12 verbs), for a total of 638 lemma pairs. 4 the judgements were collected via questionnaires distributed to a large number (> 50) of academic scholars of ancient greek. the final benchmark, agree, incorporates a mix of expert-elicited pairs and expert-assessed, machinegenerated pairs. the machine-generated items are pairs of [seed lemma -nearest neighbour], with nearest neighbours extracted from word2vec models @xcite that underwent expert judgement and were assessed as highly related. for the experiments reported in this paper, we only use the human-elicited portion of the benchmark: agree-task1. this portion can be further divided into the subset of pairs that were proposed by one expert only, and the subset of pairs that were proposed by more than one annotator, under the assumptions that the latter might be cases of a stronger relatedness, and/or higher frequency. ## road map for future work we plan a larger evaluation including more model architectures, different parameters and different evaluation metrics, with the aim of understanding the differences between model types, rather than finding the 'best' model (see also @xcite , and evaluation adequacy. more investigation is needed to understand whether the difference between count-based and predictive models trained on ancient greek lies in the quality of results (i.e., if some architectures actually return less relevant nearest neighbours), or only in the kind of relationships they capture. further experiments will also concern dependency-based embeddings. moreover, this extended study will exploit the full dataset produced for the agree benchmark, including the second part of the dataset, not used for the current evaluation. since in the second phase of the data collection the experts assigned relatedness scores to human-and machine-generated lemma pairs, these items items allows ranking the lemma pairs according to their degree of relatedness, and thus for a more nuanced evaluation. ## conclusion we presented and discussed the results of an evaluation of four distributional semantic models of ancient greek, two count-based and two predictive models. the gold standard was a subset of the agree benchmark, agree-task1, including pairs of related lemmas proposed by experts of ancient greek. the evaluation showed that count-based models achieved higher precision and recall on agree-task1, and higher precision and recall were also achieved on average when evaluating against pairs of related lemmas proposed by more than one expert. another important finding was the great difference in performance between different lemmas. we also presented a plan for a more extended evaluation, including more model architectures, parameters, and evaluation metrics. this evaluation will take into account different degrees of relatedness between lemmas and allow for a better understanding of the differences between dsms of ancient greek and of the possible impact of such differences on computational studies in ancient greek lexical semantics.