An Evaluation of Binary Comparative Lexical Complexity Models (2022)

Abstract: Identifying complex words in texts is an important first step in text simplification (TS) systems. In this paper, we investigate the performance of binary comparative Lexical Complexity Prediction (LCP) models applied to a popular benchmark dataset — the CompLex 2.0 dataset used in SemEval-2021 Task 1. With the data from CompLex 2.0, we create a new dataset contain 1,940 sentences referred to as CompLex-BC. Using CompLex-BC, we train multiple models to differentiate which of two target words is more or less complex in the same sentence. A linear SVM model achieved the best performance in our experiments with an F1-score of 0.86.

## introduction children, second language learners, or individuals suffering from a reading disability, such as dyslexia or aphasia, can find certain words hard to read, interpret, or learn @xcite @xcite @xcite . in readability and text simplification (ts) literature, these words are known as complex words. complex and non-complex words are distinguishable. statistical, morphological, and psycholinguistic features are indicative of lexical complexity @xcite . complex words are on average longer, morphologically more unique, and less frequent in general corpora, than non-complex words @xcite @xcite . with the growing popularity of distance learning platforms, the demand for new technologies that make texts more accessible for independent and remote learning has seen an exponential increase @xcite . among these technologies, are ts systems that automatically simplify texts for various target populations. the first step in ts is generally referred to as lexical complexity prediction (lcp). lcp aims to identify which words in a text are complex and therefore are in need of simplification. it has been modeled as a binary classification task @xcite , as a regression task @xcite , and more recently as a multi-class classification task @xcite . in this paper, we explore binary comparative lcp where the goal is to determine when one target word is more or less complex than another. this new type of lcp is motivated by the need for comparative prediction methods that allow for the pairwise ranking of target words based on newly available data assigned with continuous complexity values @xcite . binary comparative lcp aims to aid ts by improving the selection and ranking of substitute candidates for a particular complex word. it achieves this by allowing for more data to be generated from a finite dataset. for instance, a dataset consisting of 10,000 complex words assigned with complexity values can be converted into 100 million comparative instances, since every complex word can be compared to every other complex word. a binary comparative lcp classifier trained on this dataset can then make binary comparative judgements as part of a sorting algorithm. this lets a lexical simplification (ls) system effectively find the most appropriate simplification for any given complex word improving the efficiency of ts and other down-stream applications. we aim to determine whether binary comparative lcp is possible when attempting to differentiate the complexities of two different target words in the same sentence ( the main contributions of this paper are: ## related work traditionally, lcp comes in two forms, it is either: a). a binary classification task, known as complex word identification (cwi) @xcite @xcite , or b). a linear regression based task, simply referred to as lcp @xcite . both cwi and lcp datasets contain target words labeled with a complexity value. this complexity value is used by a machine learning (ml) model to determine the complexity of a target word. cwi assigns a binary complexity value of either 1 (complex), or 0 (noncomplex). lcp alternatively assigns a complexity value on a continuum, ranging from 0 to 1. this continuum contains multiple labels with differing complexity thresholds: very easy (0), easy (0.25), neutral (0.5), difficult (0.75), to very difficult (1) @xcite . an example is shown in other approaches have attempted to model lcp as a multi-class classification task. pintard and françois (2020) assigned six readability levels, belonging to the common european framework of reference for languages (cefr), to target words as a means of rating their complexity for second language learners. alfter (2021) trained a variety of models, including a convolutional neural network (cnn) and a recurrent convolutional neural network (rcnn), to predict the correct cefr labels of target words taken from a multitude of cerf vocabulary lists. lcp research has also included the ranking of complex words @xcite . neural regressors have been trained to identify which of two target words is more or less complex by predicting a continuous positive or negative value belonging to an inputted word pair. a positive value indicates that the first target word is more complex than the second, whereas a negative value dictates that the second target word is more complex than the first. the magnitude of the returned value also represents the degree of difference between the two target words. positive and negative values closer to +1 or -1 respectively, show that the difference between the target words' complexities is more extreme compared to those values closer to 0 ( ## models we trained a svm model given its high performance at binary cwi @xcite @xcite , a bert model @xcite per its competitive performance at lcp-2021 @xcite @xcite , and a bert + multi-layer perceptron (mlp) model @xcite to take full advantage of bert inferred contextual features as well as the wordlevel features fed into our svm model. two naive baseline models were used to evaluate the performances of our svm, bert, and bert + mlp models: a random classifier (rc) and a majority classifier (mc). we used an integrated intel uhd graphics 620 gpu to train each model. our svm and bert models were trained over 5 epochs. the train and test split of our dataset was set to 70:30% respectively. no target words were shared between the train and test sets. svm our svm model used a radial basis function (rbf) kernel and was trained on a set of well established statistical and psycholinguistic features for lcp as these have been previously found to achieve the best results @xcite . these features were word length, word frequency, syllable count, average age of acquisition (aoa), and prevalence (familiarity). word frequency being calculated in accordance to the target word's frequency in the british national corpus (bnc) @xcite , average aoa being calculated by averaging the aoas within an updated version of the living word vocabulary dataset @xcite , and prevalence being calculated in accordance to the percentage of people who knew the target word as shown in the dataset provided by @xcite . these features were obtained in regards to the target word only and were not applied to any of the target word's neighbouring words. bert after experimenting with different hyperparameters, our bert model (bert-baseuncased) was set to have a softmax activation layer, a batch size of 200, and a learning rate of 1e-5. several inputs were also experimented with that took into consideration the target word along with varying degrees of contextual information. encoding strategies adopted by the leading systems of lcp-2021 @xcite and suggested by @xcite , were then applied to these inputs and fed into our model. we encoded each input into sub-word units, otherwise known as wordpiece tokens @xcite . we then used the class identifier special token: <cls>, the separator special token: <sep>, and two custom special tokens: <@xmath0, to distinguish between two differing target words. we referred to these as (a) the target word only, (b) single context, and (c) two contexts -tw encoding strategies respectively ( bert + mlp as bert assumes full sentences to encode the correct contextual information, we use a second bert-based architecture for a fairer evaluation with the svm model. we built a bert + mlp model by feeding encoding strategies (a), (b), and (c) into our bert model and then concatenate the outputted contextual features with those features used by our svm model per @xcite . our final bert + mlp model then utilizes both set of contextual and word-level features for binary comparative lcp. ## results our bert + mlp model achieved a greater performance compared to that attained by our standalone bert model. it was found that by concatenating the contextual features outputted by our bert model from single context input (b) with those features used for our svm model, our bert + mlp model achieved a precision, recall, f1-score, and accuracy of 0.74, 0.88, 0.80 and 0.81 respectively. additional encoding strategies (a) and (c) attained lower performances, with an f1-score of 0.56 being returned by the former and 0.64 being achieved by the latter. ## analysis and discussion comparing the complexities of two target words in the same sentence allowed us to fully utilize the complex 2.0 dataset that contained multiple instances of target words in the same context. it allowed us to generate binary comparative predictions that can be used to aid substitute selection and ranking (sections 1 and 3). binary comparative lcp also allowed us to identify which words within a sentence contributed the most or the least to a context's overall complexity. therefore, by comparing the complexities of two target words in the same sentence, we were able to identify which part of a sentence is in need of priority simplification. however, modeling binary comparative lcp in this way is not without its challenges. this is reflected in our models' performance. two factors are responsible for the superior performance of our svm model in comparison to our standalone bert model: 1). contextual similarity between the two target words, and 2). small dataset size. one of the main advantages of transformer-based models, such as bert, is their ability to infer bi-directional contextual relationships between a target word and its surrounding words and then use this contextual information to make accurate predictions @xcite . however, since the two target words we are trying to compare are in the same sentence, bert's inferred contextual features in relation to target word 1 were extremely similar to those inferred for target word 2, when given encoding strategy (c) ( our svm model's word-level features of word length, word frequency, syllable count, average age of acquisition, and prevalence (familiarity) alternatively do not utilize contextual information. instead, they rely purely on the characteristics of each target word and therefore resulted in feature representations that were more dissimilar. this had the effect of making classes 0 and 1 easier to differentiate for our svm model and thus explaining its superior performance. the superiority of word-level features is further demonstrated by the performance of our bert + mlp model. encoding strategy (b) returns from our bert model contextual information related to the shared context ( our bert + mlp model performed poorly on encoding strategy (c) as it was presented from bert two inferred contextual feature representations which were near identical. encoding strategy (c) alternatively supplied only one inferred contextual feature representation from bert, which allowed our bert + mlp model to rely more heavily on its engineered word-level features. nevertheless, encoding strategy (b) still failed to surpass our bert + mlp model's performance beyond that of our svm model. this indicates that the utilization of bert's inferred contextual feature representation from encoding strategy (b), is still inferior to an svm model using word-level features. another explanation for our models' performance is our dataset's size. the complex-bc dataset contains 1,940 instances with binary labels. transformer-based models require large amounts of data to infer meaningful feature representations @xcite , whereas an svm model when trained on a set of relevant features requires less data and is also well suited for binary classification @xcite . ## conclusion and future work this paper sought to determine whether binary comparative lcp was possible when attempting to differentiate the complexities of two different target words in the same sentence. only our svm and bert + mlp models were found to be successful having achieved f1-scores of 0.86 and 0.81 respectively. this led to the conclusion that our svm and bert + mlp models benefited from more varied word-level feature representations of target word only input than in comparison to less varied contextual input. we also believe that more data is required to conduct further experimentation and achieve greater performances, especially with transformer-based models. the complex-bc dataset will be made freely available to the research community after the publication of this manuscript. we are currently working on exploring different variables that may impact the modeling of binary comparative lcp. this includes evaluating the performance of models on target words in different contexts as well as exploring a "neutral" class with words with similar complexity scores. finally, we are interested in investigating the feasibility of binary comparative lcp on languages other than english. at this point, the 5-point likert-scale annotation introduced by complex 2.0 is only available in english, however, we expect multilingual versions of complex 2.0 to become available soon enabling us to work on languages other than english.