UNIXLONG at S em E val-2020 Task 6: A Joint Model for Definition Extraction (2020)

Abstract: Definition Extraction is the task to automatically extract terms and their definitions from text. In recent years, it attracts wide interest from NLP researchers. This paper describes the unixlong team’s system for the SemEval 2020 task6: DeftEval: Extracting term-definition pairs in free text. The goal of this task is to extract definition, word level BIO tags and relations. This task is challenging due to the free style of the text, especially the definitions of the terms range across several sentences and lack explicit verb phrases. We propose a joint model to train the tasks of definition extraction and the word level BIO tagging simultaneously. We design a creative format input of BERT to capture the location information between entity and its definition. Then we adjust the result of BERT with some rules. Finally, we apply TAG_ID, ROOT_ID, BIO tag to predict the relation and achieve macro-averaged F1 score 1.0 which rank first on the official test set in the relation extraction subtask.

## introduction definition extraction is a hot research topic in natural language processing (nlp) that aims to identify terms and their corresponding definitions in unstructured text. however, it is a challenging problem to extract term-definition pairs from free and semi-structured text, especially those whose term-definition pair span crosses a sentence boundary and those lacking explicit definition verb phrases. semeval is a yearly organized nlp competition. the semeval 2020 -task 6, "defteval: extracting term-definition pairs in free text", aims to extract definition from the deft @xcite corpus, which contains roughly 7,000 sets of 3-sentence groupings extracted from textbooks of various topics from cnx.org. each sentence set reflects a context window around which the author of the original text marked a bolded word to indicate a key term. defteval consists of three tasks: 1) subtask 1: sentence classification. given a sentence, it is to classify whether the sentence contains a definition or not. this is a traditional definition extraction task. this paper describes our system and participation in all subtasks of semeval 2020 task6. the challenges of defteval include: to overcome these challenges, we incorporate several key technologies: this work is licensed under a creative commons attribution 4.0 international license. the rest of this paper is organized as follows: in section 2, we briefly introduce related work on definition extraction. in section 3 we detail our proposed system. in section 4, we analyze the results of our models. finally, we conclude our work in section 5. ## related work one stream of work on definition extraction are the rule-based approaches. some linguistic rules and patterns are well designed @xcite . most patterns are relied on the fact that the definition can be captured by the common "definitor" verb phrases such as "means", "refers to", and "is". although rule-based approaches exhibit high precision if matching the patterns, they suffer from the low recall issue. another stream of work on definition extraction are the feature engineering approaches @xcite . they try to address the low recall issue by training the machine learning models on well-designed hand-made features. however, they usually yield poor performance when transferring to new domains. recently, pre-trained language models, such as bert, xlnet @xcite , roberta, have demonstrated their effectiveness in tackling nlp tasks. for example, bert, bidirectional encoder representations from transformers, redefines the performance of 11 most common nlp tasks after finetuning. xlnet outperforms bert on 20 tasks via the idea of auto-regressive models and bi-directional context modeling. roberta refreshes many leaderboards through robustly optimizing bert via the self-supervised learning scheme. the successfulness of pre-trained models on typical nlp tasks, such as text classification and ner, motivates us to apply them in tackling the tasks in the competition, including definition extraction, entity annotation, and relation extraction. multi-task learning (mtl) is an effective learning mechanism to boost the performance of each individual task by simultaneously learning several similar tasks together to leverage the knowledge among the tasks. mtl has been demonstrated its effectiveness on many nlp tasks @xcite . @xcite for learning representations across multiple natural language understanding (nlu) tasks and obtains new state-of-the-art results on ten nlu tasks. veyseh et al. 2019 proposes a joint model for definition extraction to exploit the global structures of the input sentences as well as the semantic consistencies between the terms and the definitions. wu and he, 2019 develop an approach for relation classification by enriching the pre-trained bert model with entity information. in this competition, since the subtasks are interrelated, e.g., if the word level bio tag predicted by subtask2 contains b-definition, then the classification tag for subtask1 is 1. it naturally motivates us to explore a joint model to solve them in a whole. ## system overview we elaborate the task and our system in this section. ## experiment we detail our experimental setup and present the results in this section. we first split the training dataset into 5 groups and set the dev dataset as our 6 th group. for each unique group, we take the group as a hold out or local test dataset and set the remaining groups as the training dataset. we then fit our model on the training set and evaluate it on the local test set. we choose different base models, such as bert, xlnet, albert, roberta, xlmroberta. for different base models, we finetune the hyperparameters to fit a better result in the local test dataset. for example, in roberta, the learning rate, the batch size, the training epochs are set to 3e-5, 32, and 5, respectively. each basic model predicts 5 results on the 5 cross-validation dataset. and finally the 5 basic models get 25 results. for the joint model, we try to adjust the corresponding parameters to see the results in local test dataset. we conduct various experiment on choosing different dataset, base model and parameters and ensemble the 25 results by rank average method. finally, we achieve score 0.8077 and 0.6869 on subtask1 and subtask2 official test dataset, respectively. for the relation prediction task, we predict the root id with bert and adjust some special cases with carefully designed rules. after combining bio tag with tag id and root id, we decide the relation tag. our best result achieves 1.0 f1 score, ranking the first in this subtask. ## conclusion definition extraction is a challenging nlp task to extract definition from free text. in this work, we present our implementation to solve all subtasks of semeval task6. we propose a joint model to solve definition extraction and sequence labeling subtasks simultaneously. we apply bert to predict the relation classification task and optimize the model result with some man-made rules. in the future, we plan to explore end-to-end model for the relation extraction subtask. in addition, we can keep improving the joint model's prediction accuracy for subtask1 and subtask2, especially increasing the accuracy of those tags with few training samples.