Extending an Event-type Ontology: Adding Verbs and Classes Using Fine-tuned LLM s Suggestions (2023)

Abstract: In this project, we have investigated the use of advanced machine learning methods, specifically fine-tuned large language models, for pre-annotating data for a lexical extension task, namely adding descriptive words (verbs) to an existing (but incomplete, as of yet) ontology of event types. Several research questions have been focused on, from the investigation of a possible heuristics to provide at least hints to annotators which verbs to include and which are outside the current version of the ontology, to the possible use of the automatic scores to help the annotators to be more efficient in finding a threshold for identifying verbs that cannot be assigned to any existing class and therefore they are to be used as seeds for a new class. We have also carefully examined the correlation of the automatic scores with the human annotation. While the correlation turned out to be strong, its influence on the annotation proper is modest due to its near linearity, even though the mere fact of such pre-annotation leads to relatively short annotation times.

## introduction annotation of highly-dimensional, voluminous data is expensive, time-consuming and in addition, in case of deep-niche domains, depending on expertly trained specialists, such as linguists or medical experts. therefore it may be advantageous to organize, prioritize and provide suggestions to guide further annotation efforts efficiently. especially in a situation with a rich, constantly growing set of classes, such as it is the case with ontologies. specifically, given an already partially labeled set of examples with yet unfinished set of classes, classifier based on large language models (llms) can be leveraged to navigate the landscape of possible annotations. our showcase is an event-type ontology, the synsemclass 4.0 @xcite , populated with synonymous verbs denoting events or states. the set of events is currently dynamically evolving and encompasses classes in english, czech, german and spanish, so far limited to verbs. as any ontological resource is never complete, we have investigated various methods to facilitate efficient extension of such ontologies in two ways: adding classes for greater coverage on new texts, and adding verbs to existing classes to allow for more accurate human understanding of the classes in the ontology for a particular form of the given class expression. we suggest to achieve these by 1. examining examples with consistently low class affiliation scores across a large corpus as potential candidates for new classes; 2. on the other side of the spectrum, examining high-certainty decisions of a supervised classifier to locate highly-affiliated lemmas to a particular class, corresponding to "lowhanging fruit" for a quick manual review and confirmation of the inclusion of the lemma into the suggested class. in all cases, classifier prediction serves as guidance and the annotators are briefed to consider the suggestions as election votes. the final decision is always the annotator's, who can accept or dismiss the suggestions. the organization of this paper is as follows: sect. 2 introduces the synsemclass v4 ontology and the current state of annotations. sect. 3 describes the fine-tuned llm classifier used to generate the annotation suggestions. sect. 4 describes the manual annotations post-processing. results are presented in sect. 5 and discussed in section 6. finally, we conclude in sect. 7. we release the source code at https://github. com/strakova/synsemclass_ml . ## the ontology in our experiments, we have used the czech part of the synsemclass 4.0 foot_0 @xcite in which contextually-based synonymous verbs in various languages are classified into multilingual synonym classes according to the semantic and syntactic properties they display. there is no specific model or lexicographic theory behind building the database. however, from the linguistic point of view, the notion of synonymy used is based on the "loose" definition of synonymy by lyons and jackson @xcite , or alternatively and very closely, on both "near-synonyms" and "partial synonyms" as defined by lyons @xcite or "plesionyms" as defined by cruse @xcite ). 2 from the ontological point of view, the classes are meant to reflect different event types (concepts) and collect various information about the possible forms of expression of the event type in language. the following main basic features are distinguished in synsemclass (fig. 1 ) @xcite : in synsemclass (ssc), there exist links to e.g., vallex foot_5 for czech, framenet foot_6 and verb-net 8 for english, e-valbu 9 for german, an-cora 10 for spanish. each class member is exemplified by instances of real texts (and their translations to english) extracted from translated or parallel corpora. specifically, data is extracted from the prague czech-english dependency corpus (pcedt) 11 for czech-english, from the paracrawl corpus 12 for german-english and from the xsrl dataset 13 for spanish-english. synsemclass 4.0 includes 1200 classes (885 active after merging or deleting) with 8169 class members. all classes are annotated in czech and english, 60 of them have also german annotation. spanish is not included in the web version but is under construction @xcite . ## results this section describes the results obtained as described in sect. 3 and sect. 4. for the discussion of the various outputs, see sect. 6. ## discussion of results it is well known that trained annotators often create high-quality data, needed for many nlp applications, although their services are generally expensive.the experiment described here was designed to answer several questions: as seen from sect. 5.1 (tab. 3 ), the interannotator agreement is relatively high (0.83 on average over the two sets), but the cohen's kappa κ is low (0.51 on average over the same two sets annotated). however, the low kappa is caused by the highly skewed distribution of the decisions, 21 the most of which lead to the rejection of the assignment of the lemma to the suggested class, caused mainly by the selection of a fixed number of five suggestions per lemma regardless of the score computed by the classifier. it would be possible -by using more pairs of annotators -to optimally select the number of suggested classes (i.e., most probably between 1 and 5), but it would only be relevant for the current number of classes in the ontology. as the ontology grows, the number of rejections will be different and the optimal number of classes might change. for the size of the ontology on which it has been tested, the threshold separating the yes/no decision (with the highest uncertainty being around the average of 0-3, i.e., 1.5) seems to be around 0.3 (see fig. 3 ). however, due to the linearity of the correlation (which by itself is a positive result for the classifier-see below), it would still be necessary to provide careful manual inspection results on both sides of the threshold. the same holds for setting any thresholds at the low or high ends of the classifier score scale. in terms of annotation efficiency (providing scores to the annotators vs. not providing them), the result is largely negative. a small speedup has been observed only for a1, with a2 consuming the same time for both sets. the absolute time as recorded by the annotators per lemma (i.e., 5 times the single decisions time, which was 366 × 60 ÷ 1000 ≈ 22 sec. for a1 and 420 ÷ 1000 ≈ 25 sec. for a2) is about two minutes. this is in fact a positive finding which means that the whole set of pre-classified lemmas, as processed by the classifier (3073 lemmas) would be finished within approx. 6000 minutes (100 hours) per annotator, i.e., within 200 hours with double annotation, plus adjudication time. finally, the correlation between the automatic classifier and the human annotation is very strong. of course, the bucketing to the 0.05 interval improves the correlation (see sect. 5.3), but in any case, it seems that the classifier is able to assign the score denoting the strength of affiliation of the unknown lemma to a class with high correlation to the human annotation decisions. ## conclusions and future work as discussed in the previous section (sect. 6), the strongest result achieved in this study is the correlation between the classifier scoring buckets and the human decisions (fig. 3 , sect. 5.3). while the scores themselves, when presented to the annotators, do not seem to bring higher efficiency, the selection of the classes and their presentation to the annotators (sect. 3, sect. 4.1) result in a reasonable time for the annotation of several thousand previously unseen (unassigned lemmas) to the ontology. finally, there is no strong heuristics (for the score thresholds) that would allow to assign any unseen words to existing classes automatically -a human post-inspection and annotation is needed across the whole (or almost whole) range of scores as produced by the classifier, given the linear correlation. in the future, we plan to repeat the experiment for a larger ontology (i.e., test the effort needed for sustainable development and maintenance for such an event-type ontology when it reaches high coverage), possibly with larger lms or with some additional fine tuning given the large(r) coverage at such future time.