S em E val-2020 Task 6: Definition Extraction from Free Text with the DEFT Corpus (2020)

Abstract: Research on definition extraction has been conducted for well over a decade, largely with significant constraints on the type of definitions considered. In this work, we present DeftEval, a SemEval shared task in which participants must extract definitions from free text using a term-definition pair corpus that reflects the complex reality of definitions in natural language. Definitions and glosses in free text often appear without explicit indicators, across sentences boundaries, or in an otherwise complex linguistic manner. DeftEval involved 3 distinct subtasks: 1) Sentence classification, 2) sequence labeling, and 3) relation extraction.

## introduction definition extraction as a complex, real-world task is currently an emerging field of study. traditional definition extraction approaches mostly rely on simple, syntactically straight-forward examples with relatively little variance in vocabulary. corpora, including the wcl @xcite and ukwac @xcite , typically consist of "definition sentences" which follow a standard x is a (type) y or x, such as y syntactic structure. many also contain definitors @xcite , such as "means", "is", or "is defined by". we provide a complete review of the existing state of definition extraction in @xcite . ## data the defteval task provided data from the deft corpus for training, development and testing. introduced in @xcite , the deft corpus is currently the largest and most comprehensive corpus explicitly for definition extraction. in addition to the typical definition-type sentences discussed in the introduction, the corpus also contains sentence and "sentence windows" of wide variance in syntactic and semantic construction. sentences for the corpus were retrieved from the open source textbook website cnx.org, as well as from various 2017 sec contract filings from the publicly available us securities and exchange commission edgar (sec) database. for the defteval shared task, participants were only required to use sentences extracted from textbooks. edgar annotations were provided on request, but not required for participation in the shared task. the deft corpus provides sentence "context windows" for textbook data from cnx.org to narrow the search space for possible definitions. a context window is defined as a set of three sentences: one sentence before a sentence containing a potential term (e.g. a bold word), the main sentence with the potential term, and one sentence after. though it is certainly possible that some definitions may occur further away from the bolded mention of the term, for the purposes of the shared task, definitions that occur outside of the three-sentence window around a potential term are out of scope. the corpus contains 9 different token labels (see table1), and 5 different types of relations (see ta-ble2). this expanded annotation schema allows for a wide range of data collection, as well as a better understanding of "non-traditional" definition types (i.e. beyond the typical x is a y structure). with this tag name description term a primary term alias term a secondary, less common name for the primary term. links to a term tag. ordered term multiple terms that have matching sets of definitions which cannot be separated from each other without creating an non-contiguous sequence of tokens. e.g. x and y represent positive and negative versions of the definition, respectively referential term an np reference to a previously mentioned term tag. typically this/that/these + np definition a primary definition of a term. may not exist without a matching term. secondary definition supplemental information that may qualify as a definition sentence or phrase, but crosses a sentence boundary. ordered definition multiple definitions that have matching sets of terms which cannot be separated from each other. see ordered term. referential definition np reference to a previously mentioned definition tag. see referential term. qualifier a specific date, location, or condition under which the definition holds the scientific method is a method of research with defined steps that include experiments and careful observation. 1 occasionally, terms may appear as verb phrases or other constituent phrases (e.g. adjective phrases): many crystals and solutions rotate the plane of polarization of light passing through them. such substances are said to be optically active. 2 not including minor tokenization and sentence break fixes made during the shared task, there were no major modifications to the dataset before use in the defteval task. complete details on the dataset is discussed in @xcite . ## defteval task the defteval shared task focused on fostering an expanded understanding of definition extraction for complex data, while keeping results comparable to traditional definition extraction tasks. the shared task, which ran from 4 sept 2019 to 13 march 2020, was comprised of three subtasks: sentence classification, sequence labeling, and relation extraction. participants were provided data in accordance with the defteval timeline: defteval was administered using codalab competitions, a free, open-source platform for machine learning competitions and data challenges. instructions for participation, including details about the annotation schema and information about the evaluation process, were made available during the practice and training periods of the shared task. all data and scripts were disseminated through the deft github repository foot_2 , and linked to on the codalab landing page foot_3 . inconsistencies in the data, bug fixes, and updates to tokenization errors were made regularly throughout the training period as participants surfaced questions and potential problems with the dataset. ## shared task results over the course of the shared task competition period, 279 submissions were made through the codalab training phase, with many participants additionally developing and testing their systems locally. the evaluation periods brought 217 submissions for subtask 1, 220 for subtask 2, and 113 for subtask 3. results are based on the evaluation metrics described in section 3. participants in defteval 2020 used a various array of methods for the three subtasks. for subtasks 1 and 2 (see tables 3 and 4 ), many participants opted simply to use a pre-trained language model, especially bert, roberta, and xlnet (see @xcite , avram et al. (2020), singh et al. (2020), jeawak et al. (2020), ranasinghe et al. (2020), and davletov et al. ( 2020) ). other implementations incorporated various lstm layers in addition to the use of bert or other transformer architecture (see @xcite ). 5 * davletov-aa davletov et al. (2020) multi-task bert 6 acnlp caspa caspani et al. (2020) roberta + stochastic weight averaging 12 unixlong unixlong xie et al. (2020) joint classification and sequence labeling pre-trained model with mlp and crf layer 16 cardiff university shsabah jeawak et al. (2020) bert with two-step fine tuning 18 bertatde zhangh2020 zhang and ren (2020) bert with bilstm + attention 26 rgcl tharindu ranasinghe et al. (2020) xlnet 32 upb andrei.avram avram et al. (2020) roberta with finetuning 40 dsc iit-ism kpriyanshu256 singh et al. (2020) bert with fine-tuned language model 46 deftpunk deftpunk soboleva and kaparina (2020) fasttext and elmo embeddings with rnn ensemble 47 tükapo haemanth kannan and ponnusamy (2020) concatenated glove and on-the-fly pos embeddings with bilstm and 1d-conv + maxpool layers participants were also particularly interested in multi-task solutions for subtasks 1 and 2, to varying degrees of success (see @xcite , @xcite , ?)). @xcite used pre-trained embeddings, then "projected the contextualized embeddings generated by the roberta model in three vectors", making predictions for all three subtasks with a cohesive architecture. some "classic" neural architectures were also used in participant submissions to subtask 1, including kannan and ponnusamy (2020)'s lstm, rnn, and "hybrid" cnn solution, which leverages "the implicit local feature-extraction performed by the convolutional layers to refine the final representations passed to the reccurant layer, which accounts for global features." @xcite also used cnn and rnn methods on subtask 1, in addition to other pretrained architectures. subtask 3 submissions (see in addition to an array of architectures, defteval also sparked conversation around the annotation schema. @xcite point out that the complexity of the schema make this task, as expected, more difficult than previous definition extraction tasks. they also note that some annotations conflict with each other in the training and test sets, which may be due to the nature of the schema, as well as the complicated content of the documents annotated. we recognized this issue, and recommend that future definition extraction tasks and tasks using the deft data consider this. others mentioned they found that their initial model explorations came up short; @xcite mention that "owing to the complexity of the deft corpus" their "syntactically aware neural networks" performed lower than expected. overall, participants employed a variety of different models, especially those based on pre-trained transformer architectures for subtasks 1 and 2 and more "classic" approaches for subtask 3. participants also pointed out discrepancies and issues with the source dataset, which will be accounted for in future work. ## conclusion overall, we believe the defteval shared task was a successful first exploration of definition extraction with the deft corpus. with more than 400 submissions and 149 registered participants, definition extraction as a complex, real-world task is clearly of interest to many. we believe that the results from this shared task provide a baseline for expectations for more complicated definition extraction tasks. though this task does not test a production-ready or real-life scenario, where models may have to predict term-definition pairs in a large document or in a wider context, the task does provide an initial look at how models perform on broad, non-standard definition extraction annotations. we recommend that future iterations of this task pay close attention to the effect of the deft corpus' three sentence context windows at test time, especially as it relates to the differences between an artificially narrow search space and documents "in the wild". we also recommend that future research includes a critical examination of conflicting annotations in the deft corpus. overall, we believe defteval has surfaced questions and research directions relevant to expanding definition extraction into more complicated linguistic environments, where data is particularly messy and the natural language it stems from is particularly complex. defteval provided a launching point for the exploration of this type of problem in definition extraction, proved the field interest for such a task, and laid groundwork for future iterations of definition extraction tasks.