OYXOY : A M odern NLP Test Suite for M odern G reek (2024)

Abstract: This paper serves as a foundational step towards the development of a linguistically motivated and technically relevant evaluation suite for Greek NLP. We initiate this endeavor by introducing four expert-verified evaluation tasks, specifically targeted at natural language inference, word sense disambiguation (through example comparison or sense selection) and metaphor detection. More than language-adapted replicas of existing tasks, we contribute two innovations which will resonate with the broader resource and evaluation community. Firstly, our inference dataset is the first of its kind, marking not just one, but rather all possible inference labels, accounting for possible shifts due to e.g. ambiguity or polysemy. Secondly, we demonstrate a cost-efficient method to obtain datasets for under-resourced languages. Using ChatGPT as a language-neutral parser, we transform the Dictionary of Standard Modern Greek into a structured format, from which we derive the other three tasks through simple projections. Alongside each task, we conduct experiments using currently available state of the art machinery. Our experimental baselines affirm the challenging nature of our tasks and highlight the need for expedited progress in order for the Greek NLP ecosystem to keep pace with contemporary mainstream research.

## introduction it is a well known fact that the natural language processing world is running at multiple speeds. a select few languages claim the lion's share in the literature, boasting a plethora of models and a constant stream of results, while others are struggling to keep up with last year's state of the art. meanwhile, multilingual models, despite being heralded as the end-all solution to the issue, often fall short of expectations @xcite @xcite @xcite . the assumption that one-size-fits-all multilingual models can effectively bridge the language gap is hard to either refute or validate, given the disproportionate distribution of training and evaluation resources among languages @xcite @xcite . further muddying the waters is the dubious quality of the increasingly trending multi-and mono-lingual resources generated through minimally supervised machine translations from english @xcite . while such endeavors can certainly make for good first steps, they are neither sufficient nor without risks. the wide adoption of the practice threatens resource plurality, as more and more "new" datasets are in fact old in all but language. furthermore, it condones the accumulation of academic authority to a select few, namely the authors of the originals, promoting the unhindered perpetuation of their biases and oversights as universal across languages. worse yet, it outsources linguistic expertise to machine labor, as we are now entrusting our automated processes with capturing the nuances of under-represented languages; exactly those languages that require opinionated and targeted expert attention the most. and while a discussion on the structural causes behind the problem and the ways to incentivize change is long overdue, here we set our aims towards something more actionable. noting the striking absence of evaluation benchmarks for modern greek, and the language's limited presence in multi-lingual resources, we set out to develop a linguistically motivated and technically relevant suite of evaluation tasks. this paper aims to kickstart this endeavor, while serving as an open invitation to interested parties. concretely, we set the pace with four evaluation tasks: ## oyxoy inspired by glue and superglue @xcite , our goal is to develop a language-adapted suite that selects and extends a few key aspects of the original. our project, which we lightly dub oyxoy (pronounced /"u.xu/), is not primarily focused on offering general diagnostics, but rather on highlighting the semantic, syntactic, and morphological attributes of the greek language, and quantifying their impact on nlp systems. to that end, we present four high-level tasks that require varying degrees of lexical & sentential meaning comprehension. ## experimental baselines to quantitatively evaluate the difficulty of the tasks described in the previous section, and in order to facilitate future research in this direction, we set up some experimental baselines using the current stateof-the-art machinery available for modern greek. all our experiments rest on the tried and tested fine-tuning process for bert-like models (kenton and toutanova, 2019), using greek bert as our universal core model @xcite . ## related work nli is widely considered one of the core problems towards natural language understanding, with a plethora of evaluation suites @xcite @xcite @xcite which continue to pose significant challenge for current state-of-the-art models @xcite @xcite @xcite . like glue and super-glue, our inference examples come packed with linguistic tags to facilitate diagnostic analysis. unlike other datasets, our examples may specify more than one inference label, accounting for all possible sentence readings. at the time of writing, other than a fragment of xnli (produced by automatic translation), the only nli dataset for greek we are aware of is by @xcite (which we adapt here to our format). sense repositories, i.e., mappings between words and sets of meanings are often framed as dictionary-like structures @xcite . our dataset stands out in providing both a definition and a collection of examples for each sense, allowing the incorporation of either or both into various possible tasks and model pipelines; we show three concrete examples of how this can be accomplished. the tasks obtained, namely words-in-context, sense selection and metaphor detection, are of prime importance for the experimental validation of the lexical semantic capacities of language processing systems @xcite @xcite @xcite . to the best of our knowledge, this is the first dataset of its kind, and among the first lexical resources for greek in general. ## conclusions and future work our vision is that of an open-source, communityowned, dynamically adapted, gold-standard suite that enables the linguistically conscious evaluation of the capacities of greek language models. we have presented four novel tasks and corresponding baselines towards that goal. while our results aren't directly comparable to existing benchmarks, they do highlight the significant challenge our tasks present. this underscores the urgency for accelerated progress within the greek nlp ecosystem to stay aligned with contemporary mainstream research. pending community feedback, we hope to enrich the existing datasets by scaling them up, correcting possible artifacts and extending the language domain with regional and dialectal variations. possible tasks that we would like the project to eventually incorporate include gender bias detection, paraphrase identification, and natural language inference with explanations, among others. we are curious to continue experimenting with ways to utilize traditional resources, and exploring their potential as dataset generators for under-resourced languages in conjunction with large language models.