D eft P unk at S em E val-2020 Task 6: Using RNN -ensemble for the Sentence Classification. (2020)

Abstract: This paper describes participation in DeftEval 2020 (part of SemEval sharing task competition), and is focused on the sentence classification. Our approach to the task was to create an ensemble of several RNNs combined with fasttext and ELMo embeddings. Results show that various types of models in an ensemble give a performance boost in comparison to standard models. Our model achieved F1-score of 78% for a positive class on the DeftEval dataset.

## introduction definition extraction has been a popular topic in nlp for a long time. early research in this field dates back to the end of the 20 th century @xcite . throughout the last two decades numerous methods to solve the problem were proposed: rule-based approaches @xcite , machine learning approaches (for example, genetic algorithms @xcite , probabilistic topic models @xcite ) etc. interest in the definition extraction can be explained by the fact that it is applicable in a large number of areas, including e-learning @xcite , glossary creation @xcite , word sense disambiguation @xcite and information extraction @xcite . the goal of the task was to correctly label sentences as definitions or non-definitions. according to the organizers, most of the research in the definition extraction was based on the assumption that definitions can be retrieved by detecting keywords or syntactic structure, following the so-called aristotelian idea of the definition: "x is y, which is..." @xcite . the corpus, used for this task, is focused on other types of definitions which are harder to capture with simple pattern detection. @xcite summed up two main approaches to classification of definitions: usually the main method was either pattern matching or hypernym detection. they also noted that previous research was aimed primarily at english sources and that the number of available definition extraction corpora is quite limited. perhaps because the first research in the field was conducted relatively long ago and the open source culture was not as strong, a large number of corpora remained private. this is especially visible in espinosa-anke et al. @xcite , where seven definition extraction corpora are listed, but only two of them are easily accessible. apart from the deft corpus, used in this shared task, to our knowledge, only two other definition extraction corpora are publicly available: so-called w00 corpus @xcite and wcl corpus @xcite . unfortunately, their material mostly follows the aristotelian model of definition, hence, they were ineffectual as additional training data. it is noticeable that the problem of definition extraction has become less fundamental in recent years. for example, the state-of-the-art model, described in espinosa-anke and schockaert (2018) is one of the few examples of a neural network approach to this problem, namely, a bi-lstm, where features are learned via convolved filters @xcite . we used this approach as one of our baseline models. our main model uses an ensemble method, combining several rnns (described in details in the section 3). some classical approaches were also applied to the task (e.g., pattern matching) and experiments with data augmentation techniques were conducted. some of the resulted models were used as baselines. the corpus, used for the task, consists of 21,303 sentences, taken from freely available school textbooks. the bold typed words inside textbooks were considered as terms and it was assumed that the sentence around them is a definition. personalities and places were excluded from the corpus. the topics of the textbooks belonged to one of these categories: biology, history, physics, psychology, economics, sociology, government. for the subtask 1, we were provided with the train and dev sets, where the single sentence was labeled in the binary manner (either "0", or "1"). our data preprocessing was limited to the text tokenization and removal of tags and numbers, which did not carry any semantic meaning, such as numbers of sections or links. ## model an ensemble of models can boost the performance of single classifiers by joining their best qualities @xcite . our ensemble consists of three networks outlined below. the visualization of the full system is shown in figure 1 . two models (3.1 and 3.2) use a bidirectional lstm network @xcite , which proved to be an effective nlp tool for handling text sequences @xcite , preserving long-term dependencies, as well as context. the third model (3.3) uses a bidirectional gru @xcite instead of lstm due to its simple architecture. hyperparameters were chosen empirically, resulting in a batch size of 64, hidden neurons of 256 and the dropout of 0.3 on each lstm layer. we evaluate all models every 300 steps on the development set and after two evaluation periods without an improvement we scale the learning rate by 0.3. we also apply batch normalization before the output layer @xcite . to counter the class imbalance in the training data, we weight the loss of positive examples higher than those of negative ones. the final prediction of the ensemble is the average of the probabilities obtained from each classifier. every example that received an averaged probability over the empirically chosen threshold of 0.53 receives the label definition. all models were trained on single customer grade gpu. ## evaluation this section describes evaluation baselines. we used various types of models in attempt to show possible diversity in the performance of this task. neural_de: "neural_de" (espinosa-anke and schockaert, 2018) is the name of the already mentioned state-of-the-art model for this task. its code is freely available. we adapted it slightly for our needs, leaving the initial architecture and embedding the same, and used it on our data. svm: the svm @xcite , often used in classification problems, is one of the older classical approaches. in comparison to modern deep learning methods, it does not require a lot of data, which makes it quite competitive when dealing with low resource data. under specific circumstances it can perform even better than traditional deep learning approaches @xcite . we used bag-of-ngrams (uni-, bi-and trigrams) for text representation, resulting in the vocabulary of size 457015 items. additional ngrams of pos and dependency tags were used as features, which we extracted with spacy. 4 rule based approach: in an attempt to try a more classical approach to the definition extraction and used simple rule-based model, created with the spacy library. we had three types of patterns we were looking for: simple word constructions (as "that occurs", "refer to" or "be known", where all the verbs were matched on the lemma level), syntactic patterns, which were heavily relying on pos tags (for example, "noun" + "be" + "det" + "noun) and universal dependency based patterns. logistic regression: another baseline model was a logistic regression with a simple tf-idf embeddings. it is commonly used as a baseline for various machine learning problems. bi-lstm over character bi-lstm: a model more related to our approach is a bi-lstm network that contains one layer with a definition sentence classifier and uses character based word representations. these are created by running another bi-lstm network with 64 hidden neurons over character embeddings. the final state of the forward and backward direction are concatenated and used as word representations. due to the memory limitations, we truncate words at 50 characters. simple bi-lstm with 1 layer: a simple bi-lstm network with fasttext embeddings was used to evaluate our models against similar system for a fair comparison. therefore all parameters of this network are exactly the same as defined for our target lstm networks. simple bi-lstm with 2 layers: this model is the same as previously described network, except that it has 2 layers. ## results results are shown in the results-table. due to several bugs, caused by differences in preprocessing of test and train sets, the submitted results differ from those reported in this paper. these have been fixed and the table represents the true scores. baseline: the results show that among baseline models, almost all deep learning networks outperform classical approaches. logistic regression gave better than expected results, achieving 77% of accuracy on the test set, but ultimately reaching unsatisfactory results on the f1 score for positive label. worse performance of the rule based system could be due to instability of the structure of the definition sentences, as well as lack of rules and linguistic features. the low performance of svm can be explained by a big difference in the vocabulary of train and test sets, resulting in poor word representation of test samples. we can also observe no significant difference between the bi-lstm network with 1 and 2 layers. it is quite difficult to reason the bad performance of the bi-lstm over character embeddings, but perhaps the character network is too small and having more computational power to handle a bigger architecture could improve it. low performance of neural_de model could be explained by the fact, that the model was created with the completely different format of the definition in mind. overall, all these models could achieve much higher results with better tuning. experiment: all implemented single models outperform baseline networks that we have presented. the results demonstrate, that the presence of 2 bi-lstm layers does not guarantee the best results itself, but that additional classifiers are beneficial. out of implemented models, even though lstm is the state of the art technique, gru can compete well, if used with other well performing networks (such as elmo). bi-gru with elmo demonstrates the best results across single models, reaching 76% f1 for a positive class. finally, the ensemble of all three models reaches the highest performance across all systems, improving over bi-gru with elmo by a margin of 2% on f1 for a positive score, showing that the combination of strong models and the optimal threshold probability, can boost the overall results by joining each models' certainty about a specific decision. ## discussion considering that ensemble of several models proved to be a competitive approach to the definition extraction task, it seems logical to assume that adding more models can be beneficial too. it is frequently noted that the efficiency of the ensemble also raises via diversity of the included in it models @xcite . another seemingly promising approach is data augmentation. it is evident, that there is a lack of annotated data and some attempts in improving it could be made. we attempted some simple data augmentation for this task, but results were not satisfying. we tried the approach described by @xcite , which we modified accordingly to our task logic, substituting only adjectives, adverbs and nouns with synonyms or adding a few additional ones, trying to preserve original syntactical structure, but the resulted data was unusable. there certainly could be interesting ways to create more data in the future: for example, by incorporating knowledge graphs, following the steps of @xcite . another potential approach to the task would be usage of other deep learning models such as bert that currently achieves state-of-the-art performance for numerous other nlp tasks. ## conclusion this work describes the results of participation in a competition defteval 2020 (semeval 2020) task 6 on definition extraction. we present an ensemble of two bi-lstm models over fasttext embeddings, using additional topic and a definition word binary classifiers, with a bi-gru network over elmo. the results demonstrate that bi-lstm and bi-gru systems significantly outperform non deep learning models. we also show the benefit of using joint classifiers in comparison to having simple layers. finally, we demonstrate that an ensemble of strong single systems can improve the performance, reaching 78% f1 score for a positive class.