UU -Tax at S em E val-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation (2022)

Abstract: This paper presents our strategy to address the SemEval-2022 Task 3 PreTENS: Presupposed Taxonomies Evaluating Neural Network Semantics. The goal of the task is to identify if a sentence is deemed acceptable or not, depending on the taxonomic relationship that holds between a noun pair contained in the sentence. For sub-task 1—binary classification—we propose an effective way to enhance the robustness and the generalizability of language models for better classification on this downstream task. We design a two-stage fine-tuning procedure on the ELECTRA language model using data augmentation techniques. Rigorous experiments are carried out using multi-task learning and data-enriched fine-tuning. Experimental results demonstrate that our proposed model, UU-Tax, is indeed able to generalize well for our downstream task. For sub-task 2 —regression—we propose a simple classifier that trains on features obtained from Universal Sentence Encoder (USE). In addition to describing the submitted systems, we discuss other experiments that employ pre-trained language models and data augmentation techniques. For both sub-tasks, we perform error analysis to further understand the behaviour of the proposed models. We achieved a global F1 Binary score of 91.25% in sub-task 1 and a rho score of 0.221 in sub-task 2.

## introduction predicting the semantic relationship between words in a sentence is essential for natural language processing (nlp) tasks. deep neural language models accomplish outstanding results in multiple tasks involving semantics evaluation. the question posed by the shared task presupposed taxonomies: evaluating neural network semantics (pretens) is whether neural models can detect the taxonomic relationship between nouns, especially in scenarios where the pattern and/or the set of nouns in the sentence is previously unseen @xcite . sub-task 1 is a simpler classification task, while sub-task 2 is a more complex regression task. both sub-tasks involve datasets in english, french and italian. for each sub-task, teams are permitted three submissions. for each submission, the score is averaged over the three languages. the highest score from the three submissions is reported. we propose a series of models based on pretrained language models. we enhance the provided datasets using state-of-the-art data augmentation tools, and further increase the dataset size by employing translations. the aim of both steps is to create slightly modified versions of the sentences, such that the model can learn alternative forms of nouns and patterns. for the classification task (sub-task 1), we obtained the 3 rd place, with an f1 binary score of 91.25% averaged over the three languages. for the regression task (sub-task 2), we obtained the 5 th place, with a spearman's correlation coefficient ρ of 0.221 averaged over the three languages. sub-task 2 is markedly more difficult than subtask 1 due to sentences that can be ambiguous, such as i like dogs, but not chihuahuas; some humans will judge this sentence as acceptable, while some will not. we attempt to solve both tasks by employing data augmentation techniques in order to help the models understand variations in text. our main contributions are: (i) we devise a special development-validation split to emulate the real situation in which the model must face new words and patterns, and (ii) we combine various data augmentation tools to allow the models to learn from various versions of the training dataset. in section 2 we present the task details and some of the related work that was done previously. in section 3 we motivate our choice of models. the experiments we performed are in section 4. results and conclusions are presented in sections 5 and 6. ## background for the present task, we are provided with a list of sentences following a set of patterns, all of which have two slots for noun phrases. one such sentence might be: i don't like beer, a special kind of drink. the pattern corresponding to this sentence would be: i don't like @xcite , a special kind of @xcite . sentences are labeled according to whether the taxonomic relation between the two nouns makes sense. in sub-task 1, labels are binary; a sentence such as that shown above has a label of 1, while this sentence would have a label of 0: i like huskies, and dogs too. in sub-task 2, labels are continuous, ranging from 1 to 7; these scores are based on a seven-point likert scale, judged by humans via crowdsourcing. the same dataset is presented in english, italian and french. for subtask 1, the training and test sets consist of 5 838 and 14 556 sentences, respectively; for sub-task 2, the training and test sets consist of 524 and 1 009 sentences, respectively. there are two challenges to this dataset: (i) the test dataset is much bigger than the training dataset, and (ii) there are unseen patterns and noun pairs in the test set. the combination of these hampers the ability of machine learning (ml) models trained on the training set to generalize well to the test set. indeed, that is the aim of this task: to evaluate the ability of language models to generalize to new data when it comes to inferring taxonomies. one way to conceptualize the pretens task is to reformulate it as a taxonomy extraction task with pattern classification and distributed word representations. for a given sentence, extract the noun pair and the pattern from the sentence, and then determine if the taxonomic relation between the nouns matches the relations allowed by the pattern. this formulation is motivated by previous work in taxonomy construction that relied on various approaches ranging from pattern-based methods and syntactic features to word embeddings @xcite @xcite . as promising as this approach sounds for pretens, it involves manual labeling of the noun-pair taxonomic relations in the training set, as we are not allowed to use resources such as wordnet @xcite or @xcite . a different approach is to tackle pretens as a cross-over task between extraction of lexicosemantic relations and commonsense validation. there have been semeval tasks to extract and iden-tify taxonomic relationships between given terms (semeval-2016 task 13) @xcite , and to validate sentences for commonsense (semeval-2020 task 4, sub-task a) @xcite . the aim of the common-sense validation task is to identify which of two natural language statements with similar wordings makes sense. in the semeval-2016 task 13, approaches related to extracting hypernym-hyponym relations to construct a taxonomy involved both patternbased methods and distributional methods. taxi relied on extracting hearst-style lexico-syntactic patterns by first crawling domain-specific corpora based on the terminology of the target domain and later using substring matching to extract candidate hypernym-hyponym relations @xcite . another team designed a semi-supervised model based on the hypothesis that hypernyms may be induced by adding a vector offset to the corresponding hyponym word embedding @xcite . participants in the semeval 2020 commonsense validation task had an advantage over pretens participants: they were allowed to integrate taxonomic information from external resources such as conceptnet @xcite , which eased the process of fine-tuning the language models on the down-stream task. as an example, the cn-hit-it.nlp team @xcite and ecnu-sensemaker @xcite both used a variant of k-bert @xcite with additional data; the former injects relevant triples from conceptnet to the language model, while the later also uses conceptnet's unstructured text to pre-train the language model. other systems relied on ensemble models consisting of different language models such as roberta and xlnet @xcite . in section 3 we outline the architectures chosen to tackle the two sub-tasks of pretens. we draw on previous work, as outlined above, and provide novel combinations of datasets and algorithms to improve the performance of out-of-the box language models. ## system description the systems we propose for both pretens subtasks are based on language models. in sub-task 1 we use the electra (efficiently learning an encoder that classifies token replacements accurately) transformer @xcite , while in sub-task 2 we employ use (universal sentence encoder) @xcite . ## experimental set-up 4.1 sub-task 1: classification we implement our submitted models using simple-transformers foot_2 . all models are trained for 4 epochs with a batch size of 8; these values were determined by validation, as we explain below. the model is optimized using adamw @xcite and a linear decay learning rate schedule. the learning rate is a key aspect of the performance of a trained model. a large learning rate results in quick model convergence; however, if the learning rate is too large, it will lead to drastic updates that will trigger divergent behaviour, while training a model with a too-small learning rate might lead to an under-fitted model that gets stuck in local minima @xcite . in our two-stage model, the first stage has a lower learning rate of 3×10 -5 as opposed to the 4×10 -5 assigned in the second stage, which contains the pretens training data; this is because we want the model to learn more from the real training data than from the nlpaug-edited data. a summary of the model hyper-parameters is given in for data augmentation with nlpaug, bert base is employed to obtain the contextual word embeddings for both 'insertion' and 'substitution' operations. ## results and evaluation in this section, we analyze the performance of our submitted models in both sub-tasks. we further discuss other notable experiments that were carried out. results of the submitted models for english, french, and italian are shown in ## conclusion the limited size of the training dataset as compared to the test set made it impossible to train neural networks directly on the task. as a result, we took advantage of pre-trained language models. nonetheless, the robustness of language models is highly affected by the size and variance of the downstream task data available for fine-tuning, which causes the language model to fail to generalize. hereby, we relied upon data augmentation techniques using a two-stage fine-tuning process on electra. the first fine-tuning stage was carried out using an augmented version of the dataset, while in the second stage we used the translated versions of the provided pretens training data in addition to the original data. we ranked 3 rd out of 21 teams in sub-task 1. for the second sub-task we proposed a simple model by training an svr classifier with sentence embeddings obtained from use; we ranked 5 th out of 17 teams. as an extension for future work, both sub-tasks could greatly benefit from adversarial training, which has proven its success across various nlp tasks in improving the model robustness and generalization @xcite .