RUG -1-Pegasussers at S em E val-2022 Task 3: Data Generation Methods to Improve Recognizing Appropriate Taxonomic Word Relations (2022)

Abstract: This paper describes our system created for the SemEval 2022 Task 3: Presupposed Taxonomies - Evaluating Neural-network Semantics. This task is focused on correctly recognizing taxonomic word relations in English, French and Italian. We developed various datageneration techniques that expand the originally provided train set and show that all methods increase the performance of modelstrained on these expanded datasets. Our final system outperformed the baseline system from the task organizers by achieving an average macro F1 score of 79.6 on all languages, compared to the baseline’s 67.4.

## introduction in this paper, we describe our system and approach for the semeval 2022 pretens (presupposed taxonomies: evaluating neural network semantics) shared task (zamparelli et al., 2022). @xcite the aim of this task is to gain a better understanding of the ability of language models to recognize taxonomic relations between two words. we focus on subtask 1, which is a binary classification task in which a system should predict whether a sentence is valid or not, depending on the taxonomic word relation in a given sentence. we formulate the following research question: what are effective data generation approaches in order to improve a language model's ability to recognize appropriate taxonomic word relations? in our attempt to answer this question, we experiment with multiple approaches: adding new templates, adding new nouns from similar word lists, adding additional hyponyms, inverting templates and using a paraphrasing model to create sentence variations. we use the expanded training data to fine-tune a base english bert (devlin et al., 2019) model for the final classification task. in our approach, we incorporate all three languages for this task: english, italian and french. instead of generating additional data for each language and training separate models, we opt to train an english model and translate the italian and french sentences to english, before predicting the validity labels. we choose this approach in part because several of our data generation methods are not available for french or italian. we make use of google translate, as this is a widely used state-of-the-art general-domain translation system. our model, trained on the expanded dataset, scores an average f1 score across all languages of 79.6, which is an improvement over the 67.4 baseline score. we find that the best data expansion technique is to combine multiple approaches, where the output of one method is the input for the next. our ablation experiments show that our paraphrasing method improves scores the most. all code, data and other related files can be found in our github repository. @xcite 2 task description for the binary classification subtask, the challenge was to predict the acceptability label assigned to each sentence of the test set. the participants were provided with a training set consisting of 5,838 sample sentences and their validity labels, while the test set contained 14,556 sentences. the splits were provided in english, italian and french, the latter two being slightly adapted translations of the english dataset. these sentences exemplify constructions enforcing presuppositions on the taxonomic status of their arguments a and b, as can be seen in the following examples: i like trees, and in particular birches. 1 i like oaks, and in particular trees. 0 a sentence will only get a validity label of 1 when the taxonomic relations are compatible with the sentence construction. for this task, participants were free to use external resources, with the exception of lexical resources where semantic relationships (including taxonomic ones) are manually marked, such as wordnet (miller, 1995) or babelnet (navigli and ponzetto, 2012). however, using these lexical resources was allowed for the generation of data, which is part of our approach. the task of detecting taxonomic word relations has been tried via various approaches. from purely rule-based (hearst, 1992), to using semantic treelike resources (navigli et al., 2011), to adapting pre-trained language models (atzori and balloccu, 2020; chen et al., 2021) or creating hybrid systems (shwartz et al., 2016; ravichander et al., 2020). since this semeval task is focused on neural (language) models, we opt to use bert and focus mainly on different data generation techniques. ## system overview our main research focus is to explore the effects of different data generation approaches to expand the english training data. this data is then used to train an english bert model. the unseen italian and french sentences (to predict) from the test set are first translated into english before we feed them to the model to get their final prediction. we describe these different stages in more detail below. ## experimental setup to run our experiments, we created various combinations of our data generation techniques. the full list of experiments can be seen in as mentioned, all models were tested using our custom test set with special challenging characteristics. ## results in this section, we discuss the results on both our custom test set and the official test set. we also provide an error analysis in order to gain insight into the contributions of the different data generation methods. ## conclusion we have outlined several data generation methods and show that all methods improve the performance of a model trained on the expanded datasets compared to the original training data. especially new words, new templates and paraphrasing are effective, even by themselves. surprisingly, adding hyponyms is not that effective by itself, but combined with the previously mentioned methods it scores quite well. the final system we used to participate in the task was trained on a dataset created from those four methods. with this system, we improve the baseline set by the task organizers, placing us 14th out of 21 participants. as we have shown, some unexpected effects occur when looking at how different data generation techniques perform when combined. in future research, it might be interesting to explore this in a broader context: what can cause such differences? furthermore, since the provided dataset consisted of short and specific template-based sentences, it could be interesting to experiment with longer sentences that contain more complex constructions.