UPB at S em E val-2020 Task 6: Pretrained Language Models for Definition Extraction (2020)

Abstract: This work presents our contribution in the context of the 6th task of SemEval-2020: Extracting Definitions from Free Text in Textbooks (DeftEval). This competition consists of three subtasks with different levels of granularity: (1) classification of sentences as definitional or non-definitional, (2) labeling of definitional sentences, and (3) relation classification. We use various pretrained language models (i.e., BERT, XLNet, RoBERTa, SciBERT, and ALBERT) to solve each of the three subtasks of the competition. Specifically, for each language model variant, we experiment by both freezing its weights and fine-tuning them. We also explore a multi-task architecture that was trained to jointly predict the outputs for the second and the third subtasks. Our best performing model evaluated on the DeftEval dataset obtains the 32nd place for the first subtask and the 37th place for the second subtask. The code is available for further research at: https://github.com/avramandrei/DeftEval

## introduction definition extraction from text is a challenging research task, addressed by numerous researchers in the area of natural language processing (nlp). factual question-answering systems are possible applications that can benefit from the results of this task @xcite . as a response to this challenge, @xcite introduced the definition extraction from texts (deft) corpus, a human-annotated english dataset that contains multi-domain (e.g., biology, sociology, physics) term-definition pairs from two types of documents, free (i.e., textbook) and semi-structured text (i.e., contracts), as opposed to the domain-specific wcl dataset @xcite . in addition, a shared task was proposed by semeval-2020 which was aimed at evaluating the performance of each participant system on three subtasks defined for the deft corpus. the three subtasks are the following: subtask 1: given a labeled dataset of sentences, this subtask is to build a classifier capable of distinguishing between a sentence containing both a definition and the defined term or not. we provide two examples of sentences that contain a definition, from biology and physics domains, respectively: subtask 2: given a dataset of tokenized sentences, we aim to label each token with one of the following classes: term, alias-term, referential-term, definition, referential-definition, or qualifier. the meaning of each token can be found in the corpus description paper @xcite . subtask 3: given a dataset of tokenized sentences and the tag id of each token, our goal is to predict, for each token, the type of relationship it had with another token, and also the tag id of the token it had a relation with. this is a classical relation extraction task @xcite and the relations that must be extracted are: direct-defines, indirect-defines, refers-to, aka, and supplements. previous approaches @xcite @xcite of recognizing definition sentences mainly focused on the use of linguistic clues (e.g., "is", "means", "are", "a", or "()"). however, these studies fail to classify sentences containing definitions, where the linguistic clues are non-existent. in recent years, neural network-oriented solutions are another line of research in order to capture definitions from the text. anke and schockaert (2018) have proposed an architecture that relies on two models, convolutional neural network @xcite and bidirectional long short-term memory network @xcite . @xcite combines more advanced deep learning techniques by leveraging graph convolutional neural networks with both syntactic and semantic information. however, the existing methods for definition extraction cannot benefit from language pretrained models @xcite . motivated by their recent performances in more nlp tasks, we employ these models for solving the previously mentioned subtasks. subtask 2 is related to many nlp tasks that can be formalized as a sequence labeling problem, such as part-of-speech tagging @xcite , named entity recognition @xcite , metaphor detection @xcite , emphasis selection @xcite , keyphrase extraction @xcite , fragment-level propaganda detection @xcite , complex word identification @xcite , language identification @xcite , or cyber attacks detection in server logs @xcite . for the third subtask, we did not take into consideration the fact that we were given the set of correct tags of the second subtask (which tremendously help in the relation classification) and, thus, our approaches obtained poor results on both development and test datasets, so this subtask will not be discussed in the rest of the paper. our main contributions can be summarized as follows: ## conclusions and future developments in this paper, we have presented our solution for the semeval-2020 task 6: extracting definitions from free text in textbooks (defteval). we evaluated different state-of-the-art language models both by freezing and fine-tuning their weights. we observed that the performance of all the selected models can be significantly improved by fine-tuning their weights. through a series of experiments conducted on the development dataset, we showed that roberta significantly outperforms other language models for the two subtasks. according to the official leaderboard, we obtained the 32nd place out of 56 submissions for the first subtask and the 37th place out of 51 submissions for the second subtask. one possible direction for future work is to evaluate the multi-task scenario using other language models. we also consider that a larger annotated dataset together with the large variants of the pretrained language models could drastically improve definition extraction performance. moreover, we believe that by using class weights instead of oversampling for the first subtask, one can mitigate the problems observed in its confusion matrix.