3 D - EX : A Unified Dataset of Definitions and Dictionary Examples (2023)

Abstract: Definitions are a fundamental building block in lexicography, linguistics and computational semantics. In NLP, they have been used for retrofitting word embeddings or augmenting contextual representations in language models. However, lexical resources containing definitions exhibit a wide range of properties, which has implications in the behaviour of models trained and evaluated on them. In this paper, we introduce 3D-EX, a dataset that aims to fill this gap by combining well-known English resources into one centralized knowledge repository in the form of <term, definition, example> triples. 3D-EX is a unified evaluation framework with carefully pre-computed train/validation/test splits to prevent memorization. We report experimental results that suggest that this dataset could be effectively leveraged in downstream NLP tasks. Code and data are available at https://github.com/F-Almeman/3D-EX.

## introduction lexicographic definitions have played an important role in nlp. for example, definitions, and more specifically, term-hypernym pairs occurring in them, constitute a core component in applications such as taxonomy learning @xcite @xcite , knowledge base construction @xcite , or for augmenting language models (lms) @xcite . for this reason, numerous works have proposed methods to extract definitions from corpora (definition extraction, or de) @xcite @xcite . however, de, traditionally framed as a sentence classsification problem, plateaus quickly in terms of its applicability to real-world settings for a number of reasons, namely: (1) it is tied to a reference corpus; (2) it does not handle flexible contexts (e.g., definitional information appearing across several sentences); and (3) incorporating monolithic sentence-level definitional knowledge into lms during pretraining is not straightforward. a complementary task to the above is definition modeling (dm), a promising direction both from resource creation and nlp standpoints. dm is the task of automatically generating human-readable lexicographic definitions or glosses given some input. from its inception, where @xcite trained a bidirectional lstm on ⟨t, d⟩ pairs, where t is an input term, and d is its corresponding definition, more recent contributions in this area have leveraged contextualized representations by augmenting t with some context c @xcite @xcite @xcite . a crucial prerequisite for enabling, among others, successful dm systems is having access to datasets that combine terms, definitions, and good dictionary examples @xcite @xcite . in lexicographic resources, these good dictionary examples are written by professional lexicographers or domain experts, and often adhere to some style guidelines. this makes these sentences a valuable contextual resource for understanding the meaning of words, sometimes complementing knowledge gaps that may still exist even after reading a concept's definition. dm is, arguably, one of the most recent direct nlp application of lexical resources. we therefore argue for the need of a centralized repository that could be used to train and test dm systems, explore out-of-domain generalization, and most importantly, act as a unified test bed for lexical semantics tasks. in this paper, we fill this gap by introducing 3d-ex, a dataset that unifies a diverse set of english dictionaries and encyclopedias. our results suggest that, indeed, 3d-ex is a valuable resource for testing generative models in lexicographic contexts due to its varied sources, which makes it hard to memorize, and is also helpful for augmenting competitive baselines in downstream tasks. ## related work lexical resources have a long-standing tradition in lexical semantics @xcite . given the breadth of the area, we will review some of the most prominent existing resources, and then focus on how these resources have been leveraged in nlp tasks. ## building 3d-ex: data cleaning a prerequisite for unifying the above resources into 3d-ex, is to perform a number of preprocessing steps. this process includes: lower-casing; removing special tokens and any noisy characters such as the tab sign; removing entries where their definitions have more than 10% of non alphanumeric characters; removing entries that have null values either in words or definitions; removing entries where examples are the same as defined terms, and removing duplicate entries within each dataset or split. ## experiments and results in order to test the usefulness of 3d-ex, we perform an intrinsic set of experiments where we "stress test" the dataset for artifacts, indirect data leakage (near-synonyms), potential for memorization, etc. this, we argue, is an important step to guarantee 3d-ex can be used for testing lexical semantics models based on it. ## conclusions and future work in this paper we have introduced 3d-ex, a dataset that unifies different encyclopedias and dictionaries into one single resource. we have conducted an in-depth analysis of the dataset across several splits (random vs lexical), as well as dictionary for the future, we would like to further explore the potential of 3d-ex for downstream nlp tasks, incorporating more resources, and exploring multilingual variants. an additional avenue would be to explore the interaction of unorthodox dictionaries like urban with traditional lexicographic resources in the context of controlled technical/jargon dm. finally, leveraging 3d-ex as a resource for pretraining lms, similarly to the dictbert approach @xcite , could help inform lms with new, domain-specific and/or colloquial terms.