UPB at S em E val-2021 Task 1: Combining Deep Learning and Hand-Crafted Features for Lexical Complexity Prediction (2021)

Abstract: Reading is a complex process which requires proper understanding of texts in order to create coherent mental representations. However, comprehension problems may arise due to hard-to-understand sections, which can prove troublesome for readers, while accounting for their specific language skills. As such, steps towards simplifying these sections can be performed, by accurately identifying and evaluating difficult structures. In this paper, we describe our approach for the SemEval-2021 Task 1: Lexical Complexity Prediction competition that consists of a mixture of advanced NLP techniques, namely Transformer-based language models, pre-trained word embeddings, Graph Convolutional Networks, Capsule Networks, as well as a series of hand-crafted textual complexity features. Our models are applicable on both subtasks and achieve good performance results, with a MAE below 0.07 and a Person correlation of .73 for single word identification, as well as a MAE below 0.08 and a Person correlation of .79 for multiple word targets. Our results are just 5.46% and 6.5% lower than the top scores obtained in the competition on the first and the second subtasks, respectively.

## introduction reading is a complex process due to the mental exercise readers are challenged to perform, since a coherent representation of the text needs to be projected into their mind in order to grasp the underlying content @xcite . for non-native speakers, the lack of text understanding hinders knowledge assimilation, thus becoming the main obstacle that readers need to overcome. complex words can impose serious difficulties, considering that their meaning is often dependant on their context and cannot be easily inferred. in order to facilitate text understanding or to perform text simplification, complex tokens first need to be detected. this can be performed by developing systems capable of identifying them by individual analysis, as well as contextual analysis. there are two main approaches regarding the complexity task. tokens can be binary classified as complex or non-complex, a procedure that helps users separate problematic words from the others. words can be also labeled with a probabilistic complexity value, which in return can be used to simplify the text. words with lower degrees of complexity can be easily explained, whereas more complex tokens can be replaced with simpler equivalent concepts. the lexical complexity prediction (lcp) shared task, organized as the semeval-2021 task 1 @xcite , challenged the research community to develop robust systems that identify the complexity of a token, given its context. systems were required to be easily adaptable, considering that the dataset entries originated from multiple domains. at the same time, the target structure evaluated in terms of complexity could contain a single word or multiple words, depending on the subtask. the current work is structured as follows. the next section presents the state-of-the-art natural language processing (nlp) approaches for lcp (probabilistic) and complex word identification (cwi). the third section outlines our approaches for this challenge, while the fourth section presents the results. afterwards, the final section draws the conclusions and includes potential solutions that can be used to further improve performance. de hertog and tack (2018) introduced a deep learning architecture for probabilistic cwi. apart from the features extracted by the first layers of the network, the authors also included a series of handcrafted features, such as psychological measures or frequency counts. their architecture included different long short-term memory (lstm) modules @xcite for the input levels (i.e., word, sentence), as well as the previously mentioned psychological measures and corpus counts. sequence labeling cwi. @xcite introduced a technique based on lstms for cwi, which obtained better results on their sequence labeling task than previous approaches based only on feature engineering. the contexts detected by the lstm offered valuable information, useful for identifying complex tokens placed in sequences. changing the focus towards text analysis, @xcite extracted a series of relevant features that supports the detection of complex words. while considering their feature analysis process, the greatest influence on the overall system performance was achieved by the number of syllables and the number of punctuation marks accompanying the targeted tokens. a different approach regarding cwi was adopted by @xcite , who employed the usage of an ensemble created on the top systems from the semeval cwi 2016 competition @xcite . other experiments performed by the authors also included plurality voting @xcite , or a technique named oracle @xcite , that forced label assignation only when at least one classifier detected the correct label. @xcite tackled cwi through a cross-lingual approach. resource scarcity is simulated by training on a small number of examples from a language and testing on different languages, through zero-shot, one-shot, and few-shot scenarios. transformer-based models @xcite achieved good performance on the target languages, even though the number of training entries was extremely reduced. ## results ## discussion the entries with the largest difference when compared to the gold standard are represented by the ones that are part of the biomed category. this discrepancy is valid for both subtasks (i.e, single target word and multiple target words). the biomed entries employ the usage of more complex terminology, quantities, or specific scientific names. therefore, it becomes more difficult for standard pre-trained transformer systems, such as bert or roberta, to adapt to the biomed entries. in contrast, corpora with easier to understand language (i.e., bible and europarl) are not properly represented when using biobert or scibert, considering that the transformers are mainly pre-trained for scientific or biomedical texts. moreover, a considerable part of the biomed entries contains large amounts of abbreviations, while other entries from the same domain have ## conclusions and future work this work proposes a modular architecture, as well as different training techniques for the lexical complexity prediction shared task. we experimented with different variations of the previously mentioned architecture, as well as textual features alongside machine learning algorithms. first, we used different word embeddings and transformerbased models as the main feature extractors and, at the same time, we examined a different training technique based on adversarial examples. second, other different models were added, such as character-level embeddings, graph convolutional networks, and capsule networks. third, several hand-crafted features were also considered to create a solid baseline covering both deep learning and traditional machine learning regressors. for future work, we intend to experiment with altering the modular architecture such that the models are trained similar to a generative adversarial network @xcite , thus further improving robustness and achieving higher scores in terms of both pearson correlation coefficients and mae.