BERT at DE at S em E val-2020 Task 6: Extracting Term-definition Pairs in Free Text Using Pre-trained Model (2020)

Abstract: Definition extraction is an important task in Nature Language Processing, and it is used to identify the terms and definitions related to terms. The task contains sentence classification task (i.e., classify whether it contains definition) and sequence labeling task (i.e., find the boundary of terms and definitions). The paper describes our system BERTatDE1 in sentence classification task (subtask 1) and sequence labeling task (subtask 2) in the definition extraction (SemEval-2020 Task 6). We use BERT to solve the multi-domain problems including the uncertainty of term boundary that is, different areas have different ways to definite the domain related terms. We use BERT, BiLSTM and attention in subtask 1 and our best result achieved 79.71% in F1 and the eighteenth place in subtask 1. For the subtask 2, we use BERT, BiLSTM and CRF to sequence labeling, and achieve 40.73% in Macro-averaged F1.

## introduction definition extraction refers to extracting term-definition pairs from real texts. it is of great significance in technology and application development of information processing. because it can be applied to many scenarios. for instance, it can be used to construct domain dictionary, knowledge graph and automatic question answering system, etc. and it also can improve the efficiency of search engine to a certain extent. as we all known, nature language is always complex, for example, polysemy is a pervasive phenomenon for some words. there are two complicate aspects. first, term-definition pairs appears in different sentences. second, there are lack of explicit definitions in some sentences. we can't find obvious sign of definable words, like is, means, is defined as, etc. considering these real situations, the deft corpus @xcite collects some data that can reflect the reality. semeval-2020 task 6 @xcite and it is called by defteval for short. defteval sets up three subtasks on the deft corpus. subtask 1 is a sentence classification task. for a giving sentence, subtask 1 is used to classify whether it contains a definition. if the definition is included, the corresponding sentence is marked as '1'. subtask 2 is a sequence labeling task. for each words including punctuation in a sentence, subtask 2 is used to label them with bio tags according to the corpus' tag specification. subtask 3 is a relation classification task. given the tag sequence labels, subtask 3 is used to label the relations between each tags according to the corpus' relation specifications. in the defteval, there are seven fields, the proportion of these fields are different. and different fields have different terms. this will bring some challenges such as the boundary of terms is uncertain. as each domain has different ways to describe the domain terms. some terms appear less and may lead to the problem of out-of-vocabulary (oov) . bert can capture polysemy and context sensitive embedding, which can produce more accurate vector representation to improve the performance of system. and bert is equivalent to data enhancement, as it makes full use of a large amount of unsupervised data. so it can bring some semantic information to promote the performance of subtasks in defteval. considering that cnn is often used in image and speech recognition, it is not fully suitable for learning time series. lstm can be well used in time series prediction. and compared with bilstm, the semantic information which captured by lstm is incomplete. so we use bilstm. for subtask 1, we use bilstm and attention to further capture the semantic information of the context on the basis of bert. for subtask 2, we use bilstm and crf to sequence labeling on the basis of bert. ## related work the traditional definition extraction technologies rely on manually defined features and rules, for example, @xcite used linguistic and structural features to extract definition. @xcite used morphosyntactic patterns, automatic terminology recognition and semantic tagging with wordnet senses are used to extract definition candidates from domain-specific corpora. in recent years, a number of machine learning methods are applied in the definition extraction. for instance. @xcite used random forest that is used to identify definitions in polish documents. @xcite used weakly supervised methods that is used to extract textual definitions from naturally occurring text. a supervised approach is applied to definition extraction in which only syntactic features derived from dependency relations are used by (espinosa-anke and saggion, 2014). a joint model (veyseh et al., 2019) is applied to extract definition and its can keep syntactic connection and semantic consistency. some neural network methods are involved. for example, @xcite used neural architectures combining convolutional and recurrent neural networks and further enrich by incorporating linguistic information via syntactic dependencies. ## system overview this section we will show the details about bertatde foot_1 . first, we will introduce about the pre-trained model bert @xcite . and next we will introduce another setups about the bertatde. ## conclusion in this paper, we introduce bertatde. for multi-domain problem, we make use of bert to solve this problem. and we use bert, bilstm and attention to classify sentences, we use bert, bilstm and crf to sequence labeling. by analysing the data of subtask 2, we find the impact of imbalance between the tags. we will use multi-task learning to solve subtask 1 and subtask 2 according to the requirement of definition extraction and we will explore how to alleviate the impact of the imbalance in the future. considering that there are connections between some tags, we'll learn more about how to use the correlation between these tags to improve our model.