T ü K a P o at S em E val-2020 Task 6: Def(n)tly Not BERT : Definition Extraction Using pre- BERT Methods in a post- BERT World (2020)

Abstract: We describe our system (TüKaPo) submitted for Task 6: DeftEval, at SemEval 2020. We developed a hybrid approach that combined existing CNN and RNN methods and investigated the impact of purely-syntactic and semantic features on the task of definition extraction. Our final model achieved a F1-score of 0.6851 in subtask 1, i.e, sentence classification.

## introduction by all accounts, the first reliable english dictionary was written by samuel johnson 1 and published on 15 april, 1755. it was 18 inches tall, 20 inches wide when opened, and contained 42,773 entries. it took dr. johnson 7 years to complete, and yet it was missing the word contrafibularity, amongst others. 2 in the 265 years that have passed since, the world has evolved at a blisteringly fast pace with technology integrating itself ever more closely with our lives. however, the compilation and maintenance of dictionaries and lexicons -one of the most important and authoritative sources of meaning -continues to be the exclusive field of domain experts and lexicographers. nevertheless, with the recent advances in natural language processing, this area -like many others that deal with human language -has seen a growing interest in automating the development of such resources. definition extraction is defined as the automatic identification of definitional knowledge in text, modeled as a binary classification problem between definitional and non-definitional text. in the early days of definition extraction, rule-based approaches leveraging linguistic features showed promise. @xcite used a combination of linguistic information (n-grams, syntactic features) and structural information (position in sentence, layout) to extract definitions from dutch texts. such approaches, however, were found to be dependent on language and domain, and scaled poorly. later research incorporated machine learning methods to encode lexical and syntactic features as word vectors @xcite . @xcite tackled the problem as a language modelling task over learned definition embeddings. @xcite derive feature vectors from entity-linking sources and sense-disambiguated word embeddings. more recently, anke and schockaert (2018) use convolutional and recurrent neural networks over syntactic dependencies to achieve very good results on the wcl and w00 datasets @xcite . this paper describes our approach of combining existing methods over state-of-the-art techniques that involve the use of contextualized word embeddings such as elmo @xcite and bert @xcite in an attempt to determine if the former still offer avenues of optimization that can help them perform competitively with the latter. ## results & discussion the final model achieved a positive-class f1-score of 0.6851, ranking 47th out of 56 submissions for the first subtask. while the model under-performed in a substantial departure from our expectations, we identified factors that may have contributed to it. since the gold-standard data for the test set was we also found several incongruities in the corpus where contradictions in annotations led to an ambiguous ground-truth. consider the following sentences from the training corpus: "organisms are individual living entities." and "organelles are small structures that exist within cells." the first sentence was annotated with the positive class (contains a definition) even though the latter was not. another similar albeit more ambiguous example: "recall from the macroeconomic perspective that if exports exceed imports, the economy is said to have a trade surplus." and "if imports exceed exports, the economy is said to have a trade deficit." here, the second sentence is tagged as containing a definition even though the first is not. while some of these ambiguities can be attributed to how the training data for the binary classification task is generated from the larger sequence-annotated corpus, there are many other counter-examples where the rationale behind the annotation is unclear. such incongruities ultimately make it more challenging for the model to attain a clear and optimal generalization. ## conclusion we presented our system for definition extraction whose pre-bert methods achieved an admittedly pre-historic f1-score of 0.6851 in task 6: defteval, subtask 1. future work could potentially include the customization of the architecture to incorporate ensemble training, exploring the usage of more task-specific cues such as topical information, and perhaps even becoming one with the bert side and using contextualized word embeddings -in light of our experiments with the combination of syntactic and semantic features, the ability of the model to implicitly reproduce the classical nlp pipeline @xcite makes it a natural fit for the task. however, not all languages and domains have the ample amount of text resources required to (pre-)train large transformer-based models such as bert, not to mention the increasing computational costs of training such models. therefore, one should not lightly dismiss the advantages of linguistically-motivated, task-specific approaches in favour of more general, task-agnostic ones.