RGCL at S em E val-2020 Task 6: Neural Approaches to D efinition E xtraction (2020)

Abstract: This paper presents the RGCL team submission to SemEval 2020 Task 6: DeftEval, subtasks 1 and 2. The system classifies definitions at the sentence and token levels. It utilises state-of-the-art neural network architectures, which have some task-specific adaptations, including an automatically extended training set. Overall, the approach achieves acceptable evaluation scores, while maintaining flexibility in architecture selection.

## introduction definition extraction refers to the task in natural language processing (nlp) of detecting and extracting a term and its definition in different types of text. a common use of automatic definition extraction is to help building dictionaries @xcite , but it can be employed for many other applications. for example, ontology building can benefit from methods that extract definitions @xcite , whilst the fields of definition extraction and information extraction can employ similar methodologies. it is therefore normal that there is growing interest in the task of definition extraction. this paper describes our system that participated in two of the three subtasks of task 6 at semeval 2020 (defteval), a shared task focused on definition extraction from a specialised corpus. our method employs state-of-the-art neural architectures in combination with automatic methods which extend and clean the provided dataset. the remaining parts of this paper are structured as follows. first, we present related work in the area of definition extraction and the related field of relation extraction (section 2). the three subtasks and the dataset provided by the task organisers are described in section 3. next, we describe our system (section 4), followed by the results of the evaluation (section 5) and a final conclusion (section 6). ## related work the first efforts related to definition extraction happened in the field of hypernym extraction, where relations that usually indicate a definition were also dealt with. this includes the x is a type of y relation, such as salmon is a type of fish, where salmon is a hyponym of fish. notable work includes hearst (1992), who automatically extracts hyponyms from large amounts of unstructured text using lexico-syntactic patterns. inspired by this approach, @xcite describe a similar method to mine definitions in french, which are then classified in terms of their semantic relations, limited to the hypernymy -synonymy relation. the approach is also used for building ontologies @xcite . the importance of the semantic relations between words for pattern-based approaches to definition extraction is highlighted in @xcite . here, the authors describe and explain definitional verbal patterns in spanish, which they also propose to use for mining definitions. the proposed system is further presented in @xcite and is aimed at spanish technical texts. the system uses the aforementioned verbal patterns, as well as corresponding tense and distance restrictions, in order to extract a set of candidate terms and their definitions. once extracted, the system applies some filtering rules and a decision tree to further analyse the candidates. finally, the results are ranked using heuristic rules. all aspects of the system were developed by analysing the institut universitari de lingüística aplicada technical corpus in spanish, which is also used for evaluation. machine learning algorithms have also been used for definition extraction. gaudio and branco (2009) describe an approach that is said to be language independent and test it with decision trees and random forest, as well as naïve bayes, k-nearest neighbour and support vector machines using different sampling techniques to varying degrees of success. @xcite process polish texts and use balanced random forests, which bootstrap equal sets of positive and negative training examples to the classifier, as opposed to a larger group of unequal sets of training examples. overall, while the approach is said to increase run time, it does bring minor increases in performance with some fine-tuning. most recently, @xcite have created deft, a corpus for definition extraction from unstructured and semi-structured texts. citing some of the pattern-based approaches also mentioned here, the authors argue that definitions have been well-defined and not necessarily representative of natural language. therefore, a new corpus is presented that is said to more accurately represent natural language, and includes more messy examples of definitions. parts of the deft corpus make up the dataset for this shared task, which is described in more detail in the following section. ## subtasks and dataset the defteval shared task is split into three subtasks. the first is sentence classification, where the task is to predict whether a given sentence contains a definition. subtask 2 is a sequence labelling task, which includes requires participants to assign bio tags to indicate which tokens in a sentence belong to terms and their definitions. furthermore, the bio tags are fine-grained, denoting whether terms and definitions are primary, secondary (the second time a term or definition has been seen in a text), referential or ordered (multiple terms that have inseparable definitions). the final subtask is relation classification which requires to classify the relation between terms and their definitions. included are the tags direct and indirect definitions (links term or referential term to definition, respectively), supplements (links indirect to direct definition), refers-to (links referential term/definition to term/definition) and aka (links alias term to term). the corpus provided by the organisers is made up of parts of the deft corpus described in @xcite . this corpus has been compiled specifically for definition extraction tasks and is made up of legal contracts (2.443 sentences) and textbook data (21.303 sentences). citing a growing need for definition extraction corpora, the creators also developed an annotation scheme that is specific to the task of definition extraction. ## methodology in this section we present the different approaches we employed for each subtask. the overall approach is based on a neural network architecture, but each subtask requires different methods of preprocessing the data, as well as task-specific tweaks to the data and architecture. our implementation has been made available on github. foot_0 ## evaluation in this section we present the evaluation results that were obtained during testing. we also provide a brief look at the final submission results of the shared task. ## conclusion we have presented the system the rgcl team has prepared for the semeval-2020 task 12. the design of the system allows for easy switching of different architectures to accommodate the needs of the task at hand. for this task, we have shown the transformer architecture using xlnet is the most successful when working with limited resources. it has also been shown that data augmentation techniques we experimented, while not detrimental to overall performance, do not necessarily improve performance. in a shared task setting, the effect of the extended data from wikipedia was not useful, however, for a wider approach with higher recall, this could be more helpful. we also tried to participate in the final subtask, relation classification. however, due to time constraints, we were not able to achieve a valid submission for the this subtask. we approached it as a sequence pair classification task and employed a siamese neural network which was shown to perform well in sequence pair classification tasks @xcite . the architecture we employed is similar to the architecture presented in @xcite . when two sequences have a relation, we extracted the sequences and provided them as the input for the siamese transformer architecture. then we used the objective function suggested as classification objective function in @xcite and optimised the cross-entropy loss. due to the complexity of this task, we managed to run only a baseline of the proposed architecture which achieved very low evaluation scores on the development data. therefore, we did not have a submission for this task and do not present any results here. in future, we hope to carry out further experiments with siamese transformer architectures for relation classification tasks. going forth, we also wish to use this system for further tasks across further languages. while we may not achieve the best performance, the system utilises realistic system resources and is therefore very versatile. this is particularly with regard to the first subtask, where the difference to the best team is around 0.09, whereas for subtask two the best team is 0.36 ahead of us, indicating that our system is not competitive. it is possible to extend these experiments to a different domain easily using a pretrained transformer model in that domain given that a corpus similar to deft corpus is available in that domain. for an example, our system should be easily adoptable to biology domain using the biobert pretrained transformer model @xcite and a deft corpus like corpus on biology domain.