DSC IIT - ISM at S em E val-2020 Task 6: Boosting BERT with Dependencies for Definition Extraction (2020)

Abstract: We explore the performance of Bidirectional Encoder Representations from Transformers (BERT) at definition extraction. We further propose a joint model of BERT and Text Level Graph Convolutional Network so as to incorporate dependencies into the model. Our proposed model produces better results than BERT and achieves comparable results to BERT with fine tuned language model in DeftEval (Task 6 of SemEval 2020), a shared task of classifying whether a sentence contains a definition or not (Subtask 1).

## introduction definition extraction from free text (deft) @xcite involves finding term definition pairs from free and semi-structured texts, especially those whose term-definition span crosses a sentence boundary and those which do not have a definition phrase. an example of a cross boundary sentence from the deft corpus is given below: in doing so , <@xmath0. this type of reaction is known @xmath1, which means " <@xmath2. "(0) it can be observed that the definition (enclosed by def tags) and the corresponding terms (enclosed by term tags) are present in different sentences, thus increasing the difficulty of definition extraction. the task is thus relatively different and complex from the conventional definition extraction (de) task in which a definition could be broken into the following sub-parts: in the example presented above, the definition (rest field) and the corresponding term (definien-dum field) are present in the same sentence. moreover, the presence of definitor(s) in the text also eases the task of extracting such definitions. the order of the tags in the wcl corpus @xcite , one of the conventional corpus for definition extraction task, is predefined viz. definiendum, definitor, definiens, rest. on the other hand, there is no such predefined order for the deft corpus regarding the occurrence of bio tags. this absence of order makes it difficult for the models to identify the definitional sentences in the deft corpus and then associate the words present in the definitions with the appropriate tags. another observation is the variation in the pattern of the definitional sentences in the deft corpus due to the presence of the heterogeneous distribution of tags. some tags like 'o','i-term','i-definition', etc. are very frequent whereas tags such as 'alias-term', 'secondary-definition' are rarely seen. this imbalance in dataset causes difficulty in finding a structure for the definition and hence makes the deft corpus relatively more complex when compared to conventional corpora like wcl. thus, given the complexity of the definitions present in the deft corpus, the whole pipeline for the extraction of meaningful term-definition pairs from free text can be restructured as presented in figure 1 . since the corpus is already in the form of sentences, sentence segmentation is not required. for tagging tokens, authors of deft corpus @xcite have defined a new annotation scheme, a part of which has been touched upon in the introduction section. moreover, it can be found in the corresponding paper. in this paper, we present the approach used for the task "defteval: extracting term-definition pairs in free text" of semeval 2020. the subtasks are as follows: the rest of the paper is organized as follows: related work with its limitations has been discussed in section 2, followed by a description of the data used in section 3. the proposed methods have been elaborated in section 4 foot_1 . section 5 and 6 contains the results and error analysis respectively. section 7 concludes the paper and also includes the possible future work. ## related work the very first attempt at de was by @xcite . they used a "balanced random forest classifier" to classify definitions so as to handle the imbalance of the dataset. a major contribution to de was by @xcite , by using a generalization of word lattices to model textual definitions in the form of definiendum, definitor, definiens and rest. however, this approach is unable to generalise definitions, especially the ones that defy the conventional semantics i.e. common patterns found in definitional sentences. many existing works take advantage of linguistic features like syntactic dependencies. @xcite built defminer which is a supervised sequence labelling system using shallow parsing and dependency features. training crf (conditional random fields) with lexical, terminological, and structural features extracted from data has also been tried by @xcite . the use of linguistic features lead to promising results and observations. espinosa-anke and saggion (2014) took advantage of the subtrees in the dependency parsing of a sentence. feature representation of sentences were created using these subtrees. @xcite incorporated both linguistic and semantic features of sentence for training classifiers. sensembed is used to reveal the semantic compactness of definition containing sentences. with the advent of the deep learning era, the first attempt at de using deep learning was by @xcite . feature representation of sentences were learned using long short term memory (lstm) cells. text preprocessing such as replacing some selected words with pos tags, were carried out thus obtaining brilliant results. anke and schockaert (2018) developed syntactically aware neural networks by incorporating syntactic information (syntactic dependencies and dependency labels) along with the text sentences as input. with the help of pre-trained word embeddings, convolutional filters and bi-lstm cells, the model was capable of extracting both short range and long range dependencies from the text data. in the most recent attempt at definition extraction, veyseh et al. ( 2019 ) proposed a multi-task model to perform sentence classification and sequence labelling simultaneously. the model took advantage of the entire syntactic dependency tree rather than just dependencies, thus yielding state of the art results on the wcl dataset. ## dataset for the purpose of training, evaluation and testing, the deft corpus has been used. not only is the corpus significantly larger than the previously available corpora in the field of definition extraction, but the dataset also contains definitions from complex, human-annotated data across a variety of topics and from both free (textbook) and semi-structured (legal document) language. ## results we evaluate the performance of the three models on the validation set and the test set. the evaluation metric is f1 score on the positive class. the submissions of the test set were done using a 10-fold cross validation so as to increase the robustness of the results. the results are presented in the results show that fine tuning the language model of bert leads to improved results. the fine tuned bert achieves the greatest f1 score on the test set among the 3 models. the reason for such a performance boost can be attributed to the structure of the deft corpus. the corpus consists of long continuous segments from textbooks and legal documents. since there is a lot of text in the same context, fine-tuning the language model helps bert to get a better understanding of the domain of the corpus. ## error analysis we analyze the predictions on the validation set. for the predictions of all the models, we calculate the binary cross entropy loss for each example and then sort the examples as per descending loss values. we examine the top commonly mis-classified examples by each of the 3 models (in ## conclusion and future work we study the performance of bert on the deft corpus and tried to boost bert with explicit linguistic information (in the form of dependencies) using a text level gcn model. for further improvements, we can use bert with a fine-tuned language model as the component of the joint model. the models can also make use of a context in the form of the sentence preceding the sentence to be classified. the incorporation of a context can help models overcome the problem of misclassifying cross-sentence definitions.