U o R - NCL at S em E val-2022 Task 3: Fine-Tuning the BERT -Based Models for Validating Taxonomic Relations (2022)

Abstract: In human languages, there are many presuppositional constructions that impose a constrain on the taxonomic relations between two nouns depending on their order. These constructions create a challenge in validating taxonomic relations in real-world contexts. In SemEval2022-Task3 Presupposed Taxonomies: Evaluating Neural Network Semantics (PreTENS), the organizers introduced a task regarding validating the taxonomic relations within a variety of presuppositional constructions. This task is divided into two subtasks: classification and regression. Each subtask contains three datasets in multiple languages, i.e., English, Italian and French. To tackle this task, this work proposes to fine-tune different BERT-based models pre-trained on different languages. According to the experimental results, the fine-tuned BERT-based models are effective compared to the baselines in classification. For regression, the fine-tuned models show promising performance with the possibility of improvement.

## introduction taxonomic relations are one of the significant lexical relationships that have been used in many applications such as question answering @xcite , sentiment analysis @xcite and biomedical ontologies @xcite . in natural languages, there are many constructions that constrain the taxonomic relation between two nouns based on the order of these two nouns. for instance, given a sentence "i have a dog, not a pet". the construction "i have a ..., not a ..." implies that the taxonomic relation does not hold between "dog" and "pet". this can be seen as a presupposition imposed by the construction. however, this is not true since dogs are pets. thus, with various presuppositional constructions, validating taxonomic relations becomes more complicated in the real world. to address this issue, semeval2022-task3 presupposed taxonomies: evaluating neural network semantics (pretens) @xcite introduces the task where taxonomic relations have to be validated in different presuppositional constructions. this task proposes novel datasets in multiple languages (i.e., english, italian and french) containing sentences with different two-noun constructions. each sentence is labeled by an acceptability label for classification and an acceptability score for regression. two challenges have been raised in this task: (1) a taxonomic relation between two nouns in the sentence must be detected, and (2) the construction which embeds the two nouns must also be validated. to effectively validate taxonomic relations in such constructions, understanding the contexts or semantic meanings of these constructions are the key. many previous studies have shown that pretrained models comprise the prior knowledge of context comprehension @xcite . recently, the language mode called bert has been widely used in several tasks. the bert model can be pre-trained with the self-supervised method to generate word/token or sentence representations enriched with prior knowledge. then, they can be fine-tuned specifically for many downstream tasks including validating taxonomic relations. therefore, in this work, we adopt the pre-trained bertbased models in different languages to utilize the prior knowledge from the resources that they were pre-trained with. then, elaborating on the pretraining, we fine-tune these pre-trained models to predict the acceptability of each sentence. ## related work pre-trained language models such as glove @xcite and elmo @xcite have been widely used to generate word/sentence representations for many nlp applications. these representations have been proven to be effective since they are enriched with knowledge from the pre-training resources. recently, the transformer architecture @xcite , a multi-layer multi-head self-attention, has made the major success in nlp. with this architecture as a fundamental, a language model called bert @xcite was proposed. such model can be first pretrained with two self-supervising tasks, masked lm and next sentence prediction. after that, they can be fine-tuned with additional output layers to create new models for various downstream tasks. due to the huge success of the bert model, many language models stemming from it have been proposed. these include roberta @xcite , the more robust bert model, and distilbert @xcite , the modified bert model using knowledge distillation methods. all of these bert-based models can be pretrained on different corpora/resources depending on various purposes. previously, there are variations of the bert-based models pre-trained in different languages. mdz digital library team at the bavarian state library introduced the variations of the bert-based models pre-trained on italian and german corpora foot_0 . @xcite flaubert, pre-trained on a large french corpus consists of texts in diverse topics and writing styles. although they have been used to solve several tasks in many languages, how to use these models in validating taxonomic relations in real-world contexts still remains an open issue. ## system overview this task consists of two sub-tasks (1) a binary classification sub-task and (2) a regression subtask. for sub-task 1, each sentence in the datasets is labeled with 1 if it is acceptable and 0 otherwise. for sub-task 2, each sentence is labeled with the average score assigned by human annotators. the score is on a seven point likert-scale ranging from 1 that means "not at all acceptable" to 7 that means "completely acceptable". this paper proposes two similar approaches of fine-tuning the bert-based models for taxonomic relation classification and regression. for both sub-tasks, we select three pre-trained bertbased models that were pre-trained on three corpora with different languages. for english, we choose distilbert-base-uncased foot_1 , pre-trained on toronto book corpus and full english wikipedia. the model has 6 layers, 12 heads, and 768 embedding dimension and has 66m parameters in total. for italian, we select bert-base-italian-xxl-uncased foot_2 , the italian bert model pre-trained on texts from a recent wikipedia dump and the opus corpora collection. this model consists of 12 layers, 12 attention heads and 768 embedding dimension. the total number of parameters is 110m parameters. lastly, for french, we select flaubert-base-uncased foot_3 , pre-trained on text corpus consists of 24 sub-corpora gathered from different sources such as project gutenberg foot_4 and common crawl foot_5 . this model has 12 layers, 12 attention heads and 768 embedding dimension. the total number of parameters is 137m parameters. based on these pre-trained models, an additional layer is added in each model to fine-tune these models. each subtask has different settings for fine-tuning. sub-task 1: binary classification to fine-tune the models for sub-task 1, a fully-connected layer is added on top of the pooled output (the sequence embedding, i.e., the "[cls]" token embedding from the pre-trained model). this layer has an output size 2 and the softmax activation function. it outputs the probability of each class (1 and 0). the loss function for fine-tuning is the binary crossentropy loss. the final prediction of each sentence is made by selecting the class with the maximum probability. sub-task 2: regression similarly to sub-task 1, we add a fully-connected layer on top of the pooled output for model fine-tuning. this layer has an output size 1. this output is the predicted score for a regression task. the mean squared error loss function is used for fine-tuning this model. ## experiments we conducted experiments on the training and test sets provided by the task organizers. for each subtask, there are three training sets and three test sets for three different languages. the training set of each language has 5,837 samples while the test set has 14,560 samples. both training and test sets consist of sentences with different presuppositional ## conclusion this work proposes to fine-tune the pre-trained bert-based models to validate taxonomic relations in different presuppositional constructions. three different pre-trained bert-based models are selected and fine-tuned to perform classification and regression on three different languages, english, italian and french. according to the results, the fine-tuned models using the binary crossentropy loss for classification are effective compared to the baseline. as for the regression subtask, the fine-tuned models using the mean squared error loss for regression performed less effectively than the baseline when evaluated with spearman correlation. this might be the result of using the mean squared error loss for fine-tuning. this leaves room for improvement in fine-tuning the bertbased models for taxonomic relation regression.