Gorynych Transformer at S em E val-2020 Task 6: Multi-task Learning for Definition Extraction (2020)

Abstract: This paper describes our approach to “DeftEval: Extracting Definitions from Free Text in Textbooks” competition held as a part of Semeval 2020. The task was devoted to finding and labeling definitions in texts. DeftEval was split into three subtasks: sentence classification, sequence labeling and relation classification. Our solution ranked 5th in the first subtask and 23rd and 21st in the second and the third subtasks respectively. We applied simultaneous multi-task learning with Transformer-based models for subtasks 1 and 3 and a single BERT-based model for named entity recognition.

## introduction this work is devoted to defteval challenge @xcite held as part of semeval 2020. it was concerned with the problem of definition extraction. it has recently been a popular topic. however, there were few annotated datasets and they were often small in size @xcite or were limited to the cases when a term and its definition are in the same sentence @xcite . defteval is one of the first attempts to provide a structured multi-task dataset that can be used for various tasks connected with definition extraction and labeling @xcite ) at text level (in contrast to sentence level). all the provided data was in english. our system employs a transformer-based model trained jointly for all three tasks. for each task, we add a linear layer with dropout on top of the transformer output. it allows the system to use information about sentence classes, entities it contains, and relations between them at the same time while training. this helps to improve the results for subtasks 1,3. however, a single-task model performs better for subtask 2. our system achieved f1 score of 0.844 for the first task with the difference from the first place equal to approximately 0.03 points. for the second task, our final score was equal to 0.52 while the difference amounted to 0.32. for the third task, our f1-score was equal to 0.61 while the winning system achieved the perfect score of 1.0. although named entity information was provided for the third subtask, we did not use it and included only span information into the model. the main contribution of our work is a detailed analysis of multi-task systems performance for definition extraction, classification and named entity recognition. the results show that our approach to multi-task training might be beneficial for the sequence classification task, but it requires reconsidering for sequence labeling. our code is publicly available 1 . ## background defteval was split into three subtasks 2 : this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . the dataset contained 215 files, 80 out of them were for training, 68 for validation, and 67 for the test. these files contained 7001 text extracts with 26552 sentences and 513219 word uses. the dataset contains 29011 unique tokens. the data was provided in conll 2003-like format (see fig. 1 ). the data was from several distinct domains: biology, economics, government, history, physics, psychology and sociology. defteval was held in two phases: first, there was given the data for the first two subtasks which did not contain named entity information. then the third subtask data with named entity spans and types were revealed. ## conclusion this work is about our results in defteval challenge which was devoted to finding and classifying definitions in texts. a single transformer-based model was adopted for both tasks simultaneously. we ranked 5th in the task of sentence classification and 23rd and 21st in named entity recognition and relation classification. in this paper we describe our system and analyze the errors. figure 6 : scores for subtask 1 and subtask 2 with regard to each others weights.