ACNLP at S em E val-2020 Task 6: A Supervised Approach for Definition Extraction (2020)

Abstract: We describe our contribution to two of the subtasks of SemEval 2020 Task 6, DeftEval: Extracting term-definition pairs in free text. The system for Subtask 1: Sentence Classification is based on a transformer architecture where we use transfer learning to fine-tune a pretrained model on the downstream task, and the one for Subtask 3: Relation Classification uses a Random Forest classifier with handcrafted dedicated features. Our systems respectively achieve 0.830 and 0.994 F1-scores on the official test set, and we believe that the insights derived from our study are potentially relevant to help advance the research on definition extraction.

## introduction semeval 2020 task 6 @xcite is a definition extraction problem divided into three different subtasks, in which the objective is to find the term-definition pairs in free text, possibly spanning across sentences boundaries. in subtask 1 the system classifies a sentence as containing a definition or not. subtask 2 consists of labeling each token with their corresponding bio tags. finally, in subtask 3 the system pairs tagged entities, and labels the relation existing between them. the definition extraction from texts (deft) corpus @xcite is the dataset used for this task, and consists of human-annotated data from textbooks on a variety of subjects. definition extraction methods in the research literature fall into one of the following categories. the earlier research on the subject applies rules-based approaches, such as @xcite whose system extracts definitions from medical publications. similar approaches can also be found in @xcite as well as in @xcite . these methods find definitions by locating words such as is called, means or is, and using grammatical rules on top of that. they generally suffer from low recall, but this issue has been addressed by features-based methods. @xcite have trained a sentence classifier with features based on bag-of-words and n-grams, position of the sentences in the text and syntactic information. @xcite expands the set of features with additional linguistic and structural information, and uses a hybrid approach between a rules-based system and a machine learning classifier. the above approaches do not generalize well to new domains, as rules and handcrafted features are often relevant to specific tasks and time-consuming to create, hence the development of machine learning and deep learning models. @xcite use a long short-term memory neural (lstm) network classifier, where features are automatically created from raw input sentences and part-of-speech sequences. contextual words embeddings depend on the adjacent text, and are pretrained in an unsupervised manner to automatically learn semantic concepts before being used on different downstream tasks @xcite . @xcite apply transfer learning to their downstream token classification problem, which consists in fine-tuning the context string embeddings to a particular task without having to learn them from scratch. transformer-based architectures have been recently used with transfer learning: @xcite solve a relation extraction problem with gpt @xcite use bert @xcite embeddings for a definition extraction task. ## systems description in this section, we describe the transformers architectures used for subtask 1, as well as the random forest classifier trained with features based on entities tags and positional information to solve subtask 3. ## results & discussion the evaluation metrics chosen by the organizers for subtask 3 is the macro-average f 1 -score for the five following relations: aka, direct-defines, indirect-defines, refers-to and supplements. the classes in the data are very imbalanced, and the distribution of examples between them in the test set is the following: 37 aka, 294 direct-defines, 13 indirect-defines, 1 supplements and 13 refers-to relations. ## conclusion our system used for subtask 1 is the combination of a sentence and a token classifier, both based on a transformer architecture. in subtask 3 we use a simple random forest model built with categorical features carrying the information of the paired entities, supplemented with post-processing rules. both approaches are supervised and perform relatively well for the subtasks at hands. including the knowledge about individual bio tags may substantially improve the performance of the model for subtask 1, showing that a good token classifier is key to solve a definition extraction problem. with the deft corpus, it is sometimes difficult to make good predictions for rare classes, especially when building a token classifier. an idea to improve the models could be to use a data augmentation approach similar to the one used in subtask 1 on the examples with elements from the rare classes. one could build models specific to the different rare classes instead of one unique token classifier, or include additional information such as part-of-speech tags. also, pretraining our transformers models on domain-specific text corpus crawled from textbooks is an idea that has been considered but not implemented due to time constraints. ideally, in a definition extraction task the sequence labeling and relation extraction steps should be handled by a single model. our results on the separate subtasks suggest that improvements can be made in this direction, as classification errors of the token classifier has an impact on the relation extraction. also, the performance of our supervised models heavily relies on the important investments made to tag the entities and relations in the text corpus. research towards semi-supervised or unsupervised approaches could be fruitful as they do not depend as much on data labeling.