K a M i K la at S em E val-2022 Task 3: A l BERT o, BERT , and C amem BERT — B e(r)tween Taxonomy Detection and Prediction (2022)

Abstract: This paper describes our system submitted for SemEval Task 3: Presupposed Taxonomies: Evaluating Neural Network Semantics (Zamparelli et al., 2022). We participated in both the binary classification and the regression subtask. Target sentences are classified according to their taxonomical relation in subtask 1 and according to their acceptability judgment in subtask 2. Our approach in both subtasks is based on a neural network BERT model. We used separate models for the three languages covered by the task, English, French, and Italian. For the second subtask, we used median averaging to construct an ensemble model. We ranked 15th out of 21 groups for subtask 1 (F1-score: 77.38%) and 11th out of 17 groups for subtask 2 (RHO: 0.078).

## introduction the recognition of lexical relationships between words and the corresponding generalization has attracted increasing attention in computational linguistics. today, there already exist resources covering manually marked semantic relationships, e.g., taxonomic relations, such as the lexical database wordnet @xcite or the multilingual dictionary and semantic network babelnet @xcite . @xcite define taxonomic relations between two terms as an is-a relation. in such a relation there is a hypernym, i.e., a supertype, and a hyponym, i.e., a subtype. both the supertype and subtype are sets covering, in our task, certain semantic categories. in a relation such as animaldog, the animal is the superordinate category and the dog is the subordinate term. furthermore, those specific sets are included in a relation forming a special hierarchy @xcite . as stated by @xcite , in such an is-a relation the supertype necessarily implies the subtype, but not vice versa. semeval 2022 task 3 is a taxonomy detection and prediction task consisting of two subtasks: a binary classification and a regression task, both covering the languages english, french, and italian. we propose an approach based on the transformer-based machine learning model bert. since bert is a bidirectional model producing state-of-the-art results @xcite , we used this pre-trained model for our analysis. for the three different languages, the corresponding bert models were used @xcite @xcite . 1 2 task description in the present shared task @xcite , the taxonomic sentence structures in the given files are composed of different artificially generated constructions enforcing presuppositions. the argument nouns in the given files come from 30 semantic categories including, among others, dogs, birds, and mammals. the given word sets already show broader and narrower categories (mammals vs. dogs/birds). nevertheless, as shown in the examples in that means the sentence structures also cover comparisons where both nouns come from construction example andtoo i like teaspoons, and mugs too. butnot i like cats, but not frogs comparatives i like apples as much as cauliflower. drather i would rather have veal than salmon. except i like seafood, with the only exception of salmon. generally i like peaches, and more generally fruits. particular i like jewelry, and in particular necklaces. prefer i do not like tiramisu, i prefer broccoli. type i like parrots, not other types of birds. unlike unlike glass, pvc is often mentioned in this text. the same broader category. this has also been covered in the work by @xcite , who refers to taxonomies as a framework represented in a hierarchy where lexical counterparts or synonyms are considered. therefore, the given shared task comes with a challenge different from only recognizing the taxonomic relation-furthermore, the embedding construction allowing or disallowing the given relation had to be checked. the participants were provided with two datasets to work with the individual subtasks. the training set for subtask 1 was composed of 5,837 sentences for each language with binary labels representing 1 as an acceptable sentence and 0 as an unacceptable sentence. this subtask covers the binary prediciton of acceptability labels of each sentence given in the test set with 14,560 samples. subtask 2 consists of the prediction of an average score on a seven-point likert scale for 1,009 sentences in the test set. the original scores in the training set were annotated by humans. figure 1 shows the scores assigned to the sentences presented in the training set in subtask 2. this figure only shows the scores averaged over all annotators since per-annotator information is not available. ## system overview we used pre-trained bert @xcite networks for subtasks 1 and 2. separate models were used for each of the languages, specifically, the alberto model @xcite was used for italian, bert base uncased was used for english, and the monolingual camembert (martin et al., 2020) model was used for french. the bert models are powerful and highly versatile language models that possess the benefit of having learned good representations of the language they were trained on. this gives them a decisive edge over using models that are trained exclusively with the data provided for training as these pre-trained models will, for example, have encountered and learned representations for words that are not in the training set but are in the test set, whereas a model trained only on the training set will have trouble dealing with these unfamiliar words. as such they offer the opportunity for better generalization. the bigger challenge of these tasks was not to produce models that perform well on the limited training data but to produce models that generalize well and do not merely overfit on the provided data. to this end, standard deep learning regularization techniques such as weight decay, dropout, and model averaging were used; nonetheless, the models performed much worse on the test data than on the validation data. actually producing models that perform better at generalizing would likely have required data augmentation and/or alternative training routines. the models were not shared between subtasks 1 and 2, meaning that while the same pre-trained models were used for subtasks 1 and 2, the model used for subtask 2 was not fine-tuned for subtask 1 and vice versa. ## results unless stated otherwise, we analyzed, evaluated, and visualized the results in r (r core team, 2020) with the help of the packages caret @xcite , dplyr @xcite , ggplot2 @xcite , plyr @xcite , readr (wickham and hester, 2020), stringr @xcite , tikzdevice (sharpsteen and bracken, 2020), tm @xcite , and xtable @xcite . ## discussion our models performed considerably worse on the test dataset than on the validation set, and these differences vary across languages and constructions. while we can't determine for sure where this significant drop in performance comes from, there are some theories worth investigating. of course, one challenge (especially of the second part) of the task is the scarcity of data. subtask 1 contains 5,837 sentences in the training data per language; and 14,560 sentences in the test set. for subtask 2, there are 524 training sentences and 1,009 test instances in each language. possibly, the regression models performed unsatisfyingly simply because there wasn't enough training data avail-able. we also wanted to investigate the inherent differences between train and test data. figure 4 compares the sentence lengths of the train and test set, grouped by language. there are huge differences between the languages and, perhaps more importantly, between the train and test sets of the three languages. the french training data contained sentences of similar length to those in the test data (mean of 8.94 for the test and 9.28 for the training set), and those sentences were generally longer than those of the other two languages. the data from the italian test set seems to contain sentences of more varied lengths than the training set. the data of the second subtask show a similar discrepancy between training and test sentences. the sentences from the test data are longer on average than those from the training data in all languages. the lengths also seem to be more variable in the test set, possibly due to the higher number of sentences. the gap between the input sentences could have led to finetuning not remarkably improving the performance of the models. we furthermore looked at the distribution of different types of constructions in more detail. there were no construction labels for the training data, so we trained a simple naive bayes classifier on the tf-idf-transformed test sentences in python using pandas (pandas development team, 2022) and scikit-learn @xcite . because of the simple structure of the input sentences and a cross-validation score of 100%, we will assume the construction labels to be accurate in further discussion. figures 5 and 6 compare the distributions of the construction labels in train and test sets of subtask 1 and 2, respectively. there are once again huge differences between training and test set. notably, the training data does not contain a single unlike sentence. despite that, the models did not necessarily perform worse on this type of construction. in total, it does not seem like the distribution of construction types in the training data influenced model performance much at all. there are instances of models performing inadequately on frequent constructions in the training data, like type constructions in the first subtask. however, drather constructions were often classified correctly despite the scarcity of training sentences. ## conclusion in this work, we discussed an approach to modeling taxonomic relationships using pre-trained language models, namely alberto @xcite , bert @xcite , and camem-bert @xcite in the context of the semeval task 3 of the year 2022. the kamikla group participated in both the classification and the regression subtask. while the performance of the models was overall unsatisfying, further analysis revealed that the type of taxonomic relation that the words in a given sentence severely affected how well the models did. while the reason for this remains unclear, it might be interesting to tailor the finetuning of the bert-based model to specific constructions or combine it with a classifier that classified the input sentences according to the type of taxonomic relation. the mean is visible in white. the difference seems to be less pronounced in the french data of subtask 1, which might have contributed to the better scores. in subtask 2, the sentences seem to be generally longer in the test data than the training data in all languages.