Definition Extraction Feature Analysis: From Canonical to Naturally-Occurring Definitions (2020)

Abstract: Textual definitions constitute a fundamental source of knowledge when seeking the meaning of words, and they are the cornerstone of lexical resources like glossaries, dictionaries, encyclopedia or thesauri. In this paper, we present an in-depth analytical study on the main features relevant to the task of definition extraction. Our main goal is to study whether linguistic structures from canonical (the Aristotelian or genus et differentia model) can be leveraged to retrieve definitions from corpora in different domains of knowledge and textual genres alike. To this end, we develop a simple linear classifier and analyze the contribution of several (sets of) linguistic features. Finally, as a result of our experiments, we also shed light on the particularities of existing benchmarks as well as the most challenging aspects of the task.

## introduction definition extraction (de) is the task to extract textual definitions from naturally occurring texts @xcite . the development of models able to identify definitions in freely occurring text has many applications such as the automatic generation of dictionaries, thesauri and glossaries, as well as e-learning materials and lexical taxonomies @xcite @xcite . moreover, definitional knowledge has proven to be a useful signal for improving language models in downstream nlp tasks @xcite . the task of de is currently approached almost unanimously as a supervised classification problem, and the latest methods have demonstrated an outstanding performance, to the point of reducing the error rate to less than 2% in some datasets @xcite . however, the high performance of these models could be mainly due to artifacts in the data, and thus they may not generalize to different domains. the main aim of this paper is to analyze to what extent is possible to learn a universal definition extraction system from canonical definitions, and to understand the core differences that currently exist in standard evaluation testbeds. in particular, we propose experiments where we develop a machine learning model able to distinguish definitions with high accuracy in a corpus of canonical definitions, and later evaluate such model in different (pertaining to different domains and genres) datasets. our evaluation datasets are two, namely: the word-class lattices (wcl) dataset from @xcite , and deft, from the semeval 2020 task 6 -subtask 1 @xcite . the former provides an annotated set of definitions and non-definitions with syntactic patterns similar to those of definition sentences from wikipedia (what the authors call syntactically plausible false definitions). the latter presents a robust english corpus that explores the less straightforward cases of term-definition structures in free and semistructured text from different domains (i.e., biology, history and government), and which is not limited to well-defined, structured, and narrow conditions. we include a detailed descriptive analysis of both corpora that identifies similarities and differences between definitions and non-definitions, later used for feature selection and analysis. we come to conclusions regarding the discriminative power of certain linguistic features. interestingly, these features alone do not have a strong effect on the results, but, the combining feature sets of different nature can improve performance, even in target corpora having heterogeneous domains and non-canonical definitions. to train and first evaluate the model, we use the annotated wcl dataset. this dataset contains sentences from a sample of the wcl corpus that includes both definitions and non-definitions with syntactic patterns very similar to those found in definitions (e.g. "snowcap is unmistakable"). the syntactic patterns are simple and represent what we could refer to as canonical definitions. we will test the performance of a model trained on this dataset, and evaluate on the deft dataset, which contains a set of definitions and non-definitions from various topics such as biology, history and government. ## related work over the last years, de has received notorious attention for its applications in natural language processing, computational linguistics and computational lexicography (espinosa-anke and saggion, 2014), as it has been proven to be applicable to glossary generation @xcite , terminological databases @xcite or question answering systems @xcite , among many others. research on de has seen contributions where the task is typically proposed as a binary classification problem (whether a sentence is a definition or not), although with exceptions @xcite . de has also been studied in languages other than english, e.g., slavic languages @xcite , spanish @xcite or portuguese @xcite . many of these approaches use symbolic methods depending on manually crafted or semi-automatically learned lexico-syntactic patterns @xcite ) such as 'refers to' or 'is a'. a notable contribution to de is the word class lattices model @xcite , which explores de on the wcl dataset, a set of encyclopedic definitions and distractors, and which we use in this paper. in a subsequent contribution, espinosa-anke and saggion (2014) present a supervised approach in which only syntactic features derived from dependency relations are used, and whose results are reported higher to the wcl method. for identifying definitions with higher linguistic variability, a weakly supervised approach is presented in espinosa-anke et al. @xcite . and finally, models based on neural networks have been leveraged for exploiting both long and short-range dependencies, either combining cnns and lstms (espinosa-anke and schockaert, 2018) or bert @xcite , and which are currently the highest performing models on wcl. ## data in this section we present the datasets utilized for our analysis, namely wcl (section 3.1) and deft (section 3.2), and provide a descriptive analysis comparing both datasets (section 3.3). ## evaluation in this section we explain our experiments in definition extraction. in particular, we train a supervised model on the wcl corpus of canonical definitions, and tested on the same corpus (via cross-validation) and the deft corpus. with this experiment we aim at understanding relevant features for definition extraction and whether features from canonical definitions can be extrapolated to other domains. section 4.1 describes the experimental settings and section 4.2 presents the main results. ## conclusion and future work in conclusion, extracting definitions from texts is a challenging research task, which is highly dependant on the distribution and scope of the application. nonetheless, in this paper we have shown that a simple svm model trained on a dataset with canonical definitions using linguistic features can provide high performance while helping us understand the task better. this model has also been evaluated on a corpus with heterogeneous domains, which also provided us with insights on the qualitative difference among definitions in each setting. our descriptive analysis discovered interesting differences and similarities between definitions and non-definitions that can be used to differentiate them automatically. the inclusion of linguistic features based on our analysis improved significantly the performance of the model. as future work it would be interesting to extend the analysis to corpora of different characteristics and languages. as an straightforward application, a model with accurate performance across corpora would allow the automatic creation of dictionaries from general or specialized domains, as well as to better understand certain topics.