C ardiff U niversity at S em E val-2020 Task 6: Fine-tuning BERT for Domain-Specific Definition Classification (2020)

Abstract: We describe the system submitted to SemEval-2020 Task 6, Subtask 1. The aim of this subtask is to predict whether a given sentence contains a definition or not. Unsurprisingly, we found that strong results can be achieved by fine-tuning a pre-trained BERT language model. In this paper, we analyze the performance of this strategy. Among others, we show that results can be improved by using a two-step fine-tuning process, in which the BERT model is first fine-tuned on the full training set, and then further specialized towards a target domain.

## introduction definitions are central to the way in which humans convey knowledge about the meaning of concepts. accordingly, a large number of general and domain-specific dictionaries have been created. as new concepts emerge, and the meaning of existing concepts shifts, these dictionaries need to be updated. this continual process is traditionally carried out by linguists or domain experts, meaning that dictionaries are never fully up-to-date. in rapidly evolving scientific domains, among others, this is a clear limitation. an appealing alternative is to automatically identify and extract definitions expressed in free text. this task of extracting term-definition pairs from text corpora is known as definition extraction @xcite . early attempts to solve this task relied on rule-based methods @xcite . however, such methods are typically only able to detect explicit, direct and structured definitions, which usually contain definitor verb phrases such as means, is, is defined as. later, a large number of supervised and semi supervised machine learning models for de have been proposed @xcite @xcite . while being able to identify a wider range of definitions, these approaches cannot be adapted to new domains efficiently, as they crucially rely on carefully designed features, which might not be available, or be less effective, in the new domain. more recently, the focus has shifted to neural network based models @xcite . the method we analyze in this paper is based on fine-tuning a pre-trained bert language model @xcite . this strategy has recently proven successful across a wide range of natural language processing (nlp) tasks. in particular, we focus on semeval-2020 task 6: defteval: extracting term-definition pairs in free text @xcite . we participated in subtask 1: sentence classification. this task required participants to predict whether a given sentence contains a definition. the associated dataset contains documents from seven different domains, including biology, history and economics. in our analysis, we focus on comparing two different strategies for fine-tuning the pre-trained bert model: domains. the second strategy avoids confusing the classifier with different definition styles, but it implies that only a limited amount of training data is available for each domain. we also experiment with a third approach, which is aimed at combining the best of both worlds: 3. fine-tuning a domain-specific bert model in two steps, by first fine-tuning the model on all training data, and subsequently specializing it to the target domain in an additional fine-tuning step. ## data we used the deft corpus @xcite , which was made available as part of the semeval-2020 task 6 competition. this dataset was collected from a repository of english open source textbooks 1 and annotated by five annotators using the brat annotation framework 2 . for subtask 1, the available sentences are split into 17,819 training sentences, 872 development sentences and 859 testing sentences. each split contains data from 7 domains: biology, history, physics, psychology, economic, sociology, and government. the number of sentences in each split for each domain are listed in ## methods given the success of bert @xcite across a wide range of nlp tasks, we decided to focus on analyzing its performance in the context of definition extraction. we considered the following variants. fine-tuning strategies. we experimented with the bert-base model, using the pytorch huggingface implementation named bertforsequenceclassification 3 . essentially, this method corresponds to adding a classification layer on top of the pre-trained bert model, and fine-tuning the bert model while training the classification layer. we will specifically compare the following fine-tuning strategies: as a baseline strategy, we also explored the following variant: ## results and discussion for all experiments, we used google colab free gpu foot_4 to train 'bert-base-uncased' models for 4 epochs. for the bert-double method, we used 4 epochs for each of the two fine-tuning steps. we set the batch size to 16 and we pad the sentences to the 256 sequence length, which gave the best performance for the bert-all model based on the development set, as shown in figure 1 . we used the adam optimizer and a learning rate of 2 • 10 -5 and 10 - foot_2 for fine-tuning bert models and lstm based models respectively. the results of the considered methods are summarized in ## conclusions we have described our participation in semeval-2020 task 6 on extracting definitions from free text in textbooks. in particular, we participated in subtask 1, where the aim was to classify a given sentence as definitional or not. we evaluated to use of lstms and we compared different strategies for fine-tuning a pre-trained bert language model. we found the latter to be more effective, especially when we fine-tuned the model twice. in particular, the bert model is then first fine-tuned on the full training set and then fine-tuned further towards the target domain.