Defx at S em E val-2020 Task 6: Joint Extraction of Concepts and Relations for Definition Extraction (2020)

Abstract: Definition Extraction systems are a valuable knowledge source for both humans and algorithms. In this paper we describe our submissions to the DeftEval shared task (SemEval-2020 Task 6), which is evaluated on an English textbook corpus. We provide a detailed explanation of our system for the joint extraction of definition concepts and the relations among them. Furthermore we provide an ablation study of our model variations and describe the results of an error analysis.

## introduction definition extraction (de) is a subfield of information extraction that deals with the automated extraction of terms and their descriptions from natural language text. we belief that for humans definitions are one of the most important sources of knowledge to clarify unknown words in a new language or domain. thus, an important application is the automated construction of a glossary, which is a laborious task if done manually. another important application is the use of definitions as background knowledge for machine learning algorithms. recent results have disclosed difficulties of state of the art approaches when confronted with factual knowledge @xcite , and have lead to a surge of research in this area @xcite @xcite . the advantages of definitions as a knowledge source are two-fold: firstly, definitions are easier to extract and annotate than other forms of knowledge sources, e.g. knowledge graphs (kg). and secondly, definitions can be processed using the same text-based algorithms whereas the use of a kg requires other forms of algorithms, e.g., graph embeddings. while research using definitions exists, it is limited by the scale of available annotations and restricts large scale applications such as in self-supervised learning setups. this paper describes our submission to the defteval challenge subtask 2. our approach is based on a multi-task learning strategy, that jointly extracts concepts and relations between them, and achieves a score of 49.68 macro f1, which ranked our best system in place 27 of 51 participants. the major challenge turned out to be the strong label imbalance that made an evaluation using the official macro-averaged f1 score difficult. the source code of our approach is publicly available. 1 ## defteval challenge the defteval challenge 2 @xcite includes three subtasks: (1) classification of definition sentences, (2) sequence labeling of definition concepts and (3) relation extraction between concepts. the deft corpus @xcite is used for evaluation, which consists of english textbooks scraped from an online-learning website. 3 compared to the previous manually annotated datasets wcl @xcite and w00 @xcite , the deft corpus is significantly larger and the examples are more diverse than simple is-a patterns. in contrast to wcl and w00 the deft corpus examples are not limited to definition sentences. instead, they consist of windows of up to three sentences around a highlighted ## model we regard the de task as the joint task of named entity recognition (ner) and relation extraction (re), and propose a system that closely follows the joint learning approach of @xcite . the ner part of their model is based on a bilstm crf tagger architecture @xcite . the re part projects each encoded token into a head and a tail space and classifies relations between all combinations of heads and tails, allowing for each token to have multiple relation heads. crf tagger in our model each input example is first passed through a bert model. the bert layers are averaged for each token using a vector of learned weights, resulting in a token embedding. after passing through bert, this token embedding is concatenated with the auxiliary input features and passed through the n bilstm layers to learn a task-specific contextualized encoding h. this is projected into logits using a fully-connected layer and a crf layer is trained to extract subtask 2 labels. this subcomponent forms the crf tagger baseline and the simple tagger baseline is the same model without the crf layer. relation extraction re is used as an auxiliary objective. as the challenge provided files in the deft format, which is a tab-separated format with a single relation tail per token, we modeled the constraint that every token could point to at most one relation tail. for each token t i , the goal is to predict an output index c i . the special case c @xmath0, if no relation exists between t i and any of the potential relation tails t j . in all other cases the index c i is decoded to the k-th relation r k between token t i and token t j . the re subcomponent uses only the hidden representation h as input. 4 the hidden representation h is passed through two projection layers u and w to extract representations for head and tail, respectively. for each token combination, the sum of the representations is passed through a relu activation function in order to create a tensor of feature activations: the class logits are computed as follows: the joint model is then trained on the sum of the losses for the two subtasks. we evaluated differently weighted losses, but found an equally-weighted sum to work best with respect to the subtask 2 performance. auxiliary input features additional input information augments the raw token input. string labels such as part of speech (pos) tags and ner labels are converted into vectors using a randomly-initialized learned embedding matrix. for tokens that participate in a coreference cluster a binary indicator variable is added. another set of binary variables indicates matches of rule-based patterns where each pattern match results in an activated indicator variable. every binary variable is represented as one dimension of the additional input. ## experiments the challenge provided labeled files split into training and development, while keeping the final test set hidden from the participants. we treated the official development set as our test set and created a new development set by randomly selecting 8 files from the training set. the resulting dataset sizes are listed in hyperparameter tuning was performed on the dev set using the allentune implementation @xcite . the source code includes config files for the conducted experiments. ## results