Document-Level Definition Detection in Scholarly Documents: Existing Models, Error Analyses, and Future Directions (2020)

Abstract: The task of definition detection is important for scholarly papers, because papers often make use of technical terminology that may be unfamiliar to readers. Despite prior work on definition detection, current approaches are far from being accurate enough to use in realworld applications. In this paper, we first perform in-depth error analysis of the current best performing definition detection system and discover major causes of errors. Based on this analysis, we develop a new definition detection system, HEDDEx, that utilizes syntactic features, transformer encoders, and heuristic filters, and evaluate it on a standard sentence-level benchmark. Because current benchmarks evaluate randomly sampled sentences, we propose an alternative evaluation that assesses every sentence within a document. This allows for evaluating recall in addition to precision. HEDDEx outperforms the leading system on both the sentence-level and the document-level tasks, by 12.7 F1 points and 14.4 F1 points, respectively. We note that performance on the high-recall document-level task is much lower than in the standard evaluation approach, due to the necessity of incorporation of document structure as features. We discuss remaining challenges in document-level definition detection, ideas for improvements, and potential issues for the development of reading aid applications.

## introduction automatic definition detection is an important task in natural language processing (nlp). definitions can be used for a variety of downstream tasks, such as ontology matching and construction @xcite , paraphrasing @xcite , and word sense disambiguation @xcite . prior work in autextual entailment is the task of determining whether a "hypothesis" is true, given a "premise". a bilm combines both a forward and backward lm [...] a fine grained word sense disambiguation (wsd) task and a pos tagging task. definition detection is especially important for scholarly papers because they often use unfamiliar technical terms that readers must understand to properly comprehend the article. in formal terms, definition detection is comprised of two tasks: classifying sentences as containing definitions or not, and identifying which spans within these sentences contain terms and definitions. as the performance of definition extractors continues to improve, these algorithms could pave the way for new types of intelligent assistance for readers of dense technical documents. for example, one could envision future interfaces that reveal definitions of jargon like "bilm" or the symbol "s despite recent advances in definition detection, much work remains to be done before models are capable of extracting definitions with an accuracy appropriate for real-world applications. the first challenge is one of recall: existing systems are typically not trained to identify all definitions in a document, but rather to classify individual sentences arbitrarily sampled from a large corpus. the second challenge is one of precision: the state of the art misclassifies upwards of 30% of sentences @xcite . this begs the questions of why definition extractors fall short, and how these shortcomings can be overcome. in this paper, we contribute the following: in summary, this paper draws attention to the work yet to be done in addressing the task of document-level definition detection for scholarly documents. we draw attention to the fact that a seemingly straightforward task like definition detection still poses significant challenges to nlp, and that this is an area that needs more focus in the scholarly document processing community. ## related work definition detection has been tackled in several ways in prior research. the traditional rule-based systems @xcite @xcite used hand-written definition patterns (e.g., "is defined as") and linguistic features (e.g., pronoun, verb, punctuation), providing high precision but low recall detection. to address the low recall problem, model-driven approaches @xcite @xcite were developed using statistical and syntactic features such as bagof-words, sentence position, part-of-speech (pos) tags, and their combination with hand-written rules. notably, @xcite used conditional random field (crf) @xcite to predict tags of each token in a sentence such as term for term tokens, def for definition tokens, and o for neither. recently, sophisticated neural models such as convolutional networks (espinosa-anke and schockaert, 2018) and graph convolutional networks (veyseh et al., 2020) have been applied to obtain better sentence representations in combination with syntactic features. however, our analysis found that the state-of-the-art is still far from solving the problem, achieving an f1 score of only 60 points on a standard test set. ## error analysis of the leading system in order to inform our efforts to develop a more advanced system, we performed an in-depth error analysis of the results of the current leading approach to definition and term identification, the joint model by @xcite . we analyzed the models' predictions on the w00 dataset @xcite since it matches our target domain of scholarly papers and is the dataset that the joint model was evaluated on. of the 224 test sentences, 1 the veyseh et al. (2020) system got 111 correct. the first author annotated the remaining 113 sentences for which the algorithm was partially or fully incorrect to ascertain the root causes of the errors. we discovered four (for terms) and five (for definitions) major causes for the erroneous predictions, as summarized in a [graph -based operator] defines a transformation on a multi-document graph (mdg) g which preserves {some of its properties while reducing the number}. . . the inductive logic programming learning method that we have developed enables us to automatically extract from a corpus n -v pairs whose elements axe linked by one of the semantic relations defined in the qualia structure ... for instance, in the bottom-most example in top hypothesized causes of error (%) overgeneralization: technical term bias 48.6% unfamiliar or unseen vocabulary 25.7% complicated sentence structure 12.9% entity detection 4.3% overgeneralization: technical term bias 28.9% overgeneralization: surface pattern bias 23.3% unseen patterns 14.4% complicated sentence structure 12.2% overgeneralization: description 3.3% we rank the causes of errors by frequency and summarize the results in proposed error correction solution types (%) syntactic (pos, parse tree, entity, acronym) 29.2% heuristics 23.6% better encoder/tokenizer, unk 18.0% rules (surface patterns) 11.3% annotations* 9.4% pattern generalization 5.7% mathematical symbol detection 1.9% more context 0.9% we again rank the error correction solutions by frequency ( ## definition sentence detection model to address the errors identified in §3, we designed heddex, a new sentence-level definition detection model. the model incorporates a set of syntactic features, heuristic filters, and encoders. each of these was designed to address a common class of error revealed in the error analysis. the model achieves superior performance over the state of the art for the task of sentence-level definition detection. ## document-level definition detection although heddex attains reasonable performance on individual sentences, it faces new challenges when applied to the scenario of document-level analysis. in this section, we evaluate sentence detection for full papers in two novel ways. first, we assess the precision of the heddex model across all of the sentences of 50 documents in §5.1. second, we assess both the precision and the recall of the algorithm across all of the sentences of 2 full documents ( §5.2, s5.3). ## discussion detecting definitions is a very challenging task, and it is far from solved. here we discuss remaining challenges and ideas for improvements, and motivate the need for high-precision, high-recall definition detection in an academic document reading aid application. outstanding technical challenges include: potential ideas for improvements of the system include: document structure and positional information may improve detection. for instance, the section information of a term would be an important feature to recognize whether a term is first introduced or not. ## conclusion this work sets the stage for bridging the gap between a well-known nlp task; definition detection, and real-world applications of the technique that requires both high precision and high recall. to achieve the goal, we proposed a more realistic setup for definition detection task called documentlevel definition detection that requires high recall, mathematical symbol recognition, and documentlevel feature engineering. our proposed definition detection system heddex achieved significant gains in both sentence-level and document-level tasks. yet, the problem is far from being solved. we suggest that better coverage of variability of expression, recognition of mathematical symbols and notation, and other nuances of the task must still be addressed.