B ig G reen at S em E val-2021 Task 1: Lexical Complexity Prediction with Assembly Models (2021)

Abstract: This paper describes a system submitted by team BigGreen to LCP 2021 for predicting the lexical complexity of English words in a given context. We assemble a feature engineering-based model with a deep neural network model founded on BERT. While BERT itself performs competitively, our feature engineering-based model helps in extreme cases, eg. separating instances of easy and neutral difficulty. Our handcrafted features comprise a breadth of lexical, semantic, syntactic, and novel phonological measures. Visualizations of BERT attention maps offer insight into potential features that Transformers models may learn when fine-tuned for lexical complexity prediction. Our ensembled predictions score reasonably well for the single word subtask, and we demonstrate how they can be harnessed to perform well on the multi word expression subtask too.

## introduction lexical simplification (ls) is the task of replacing difficult words in text with simpler alternatives. it is relevant in reading comprehension, where early studies have shown infrequent words lead to more time spent by a reader fixated on it, and that ambiguity in a word's meaning adds to comprehension time @xcite . complex word identification (cwi) is believed to be a fundamental step in the automation of lexical simplification @xcite . early techniques for conducting cwi suffer from a lack of robustness, from simplifying all words to then study its efficacy @xcite , to applying thresholds on features like word frequency @xcite . this year's lexical complexity prediction (lcp) shared task @xcite forgoes the treatment of word difficulty as a binary classification task @xcite and instead measures degree of complexity on a continuous scale. this choice is intriguing as it mitigates a dilemma with previous approaches of having to treat words extremely close to a decision boundary (suppose a threshold deems a word's difficulty) identically to those that are far away, ie. extremely easy or extremely difficult. teams are asked to submit predictions on unlabeled test sets for two subtasks: predicting on english single word and multi word expressions (mwes). for each subtask, biggreen presents a machine learning-based approach that fuses the predictions of a feature engineering-based regressor with those of a feature learning-based deep neural network model founded on bert @xcite . our code is made available on github. 1 ## related work previous studies have looked at estimating the readability of a given text at the sentence-level. mc @xcite regresses the number of polysyllabic words in a given lesson against the mean score for students quizzed on said lesson, yielding the smog readability formula. @xcite offer a list of 768 (later updated to 3,000) words familiar to grade-school students in reading, which they find correlates with passage difficulty. an issue with traditional readability metrics seems to be the loss of generality at the word-level. @xcite tries a brute force approach where a simplification algorithm is applied to each word of a given text, deeming a word complex only if it is simplified. however, this suffers from the assumption that a non-complex word does not require further simplification. they also try assigning a familiarity score to a word, and determining whether the word is complex or not by applying a threshold. we avoid thresholding our features in this study as we find it unnecessary, since raw familiarity scores can be used as features in regression-based tasks. results from cwi at semeval-2016 @xcite suggest vote ensembling predictions of the best performing models as an effective strategy, while several top-performing models @xcite @xcite appear to use linguistic information beyond just word frequency. this inspires our use of ensemble techniques, and a foray into phonological features as a new point of research. results from cwi at semeval-2018 show feature engineeringbased models outperforming deep learning-based counterparts, despite the latter having generally better performances since semeval-2016. @xcite present complex, a novel dataset in which each target expression (a single word or two-token mwe) is assigned a continuous label denoting its lexical complexity. each label lies in range 0-1, and represents the (normalized) average score given by employed crowd workers who record an expression's difficulty on a 5-point likert scale. we define a sample's class as the bin to which its complexity label belongs, where bins are formed using the following mapping of complexity ranges: [0, 0.2) → 1, [0.2, 0.4) → 2, [0.4, 0.6) → 3, [0.6, 0.8) → 4, [0.8, 1] → 5. target expressions in complex have 0.395 average complexity and 0.115 standard deviation, reflecting an imbalance in favor of class 2 and 3 samples. ## biggreen system & approaches in this section, we overview features fed to our feature engineering-based model, as well as training techniques for the feature learning-based model. we describe our features in detail in appendix a. note that fitted models for the single word subtask are then harnessed for the mwe subtask. ## results we present the performances of biggreen's system on each subtask in tables 2 and 3 . for the mwe subtask, performances are reported in ## conclusion in this paper, we report inspirations for a system submitted by biggreen to lcp sharedtask 2021, performing reasonably well for the single word subtask by adapting ensemble methods upon feature engineering and feature learning-based models. we see potential in future deep learning approaches, acknowledging the need for complementary word frequency-based handcrafted features for the time being. we surpass our submitted results for the mwe subtask, by utilizing the predictive capabilities of our single word subtask models. avenues for improvement include better data aggregation, as a relative lack of class 4,5 samples seems to hurt pearson correlation across extremely complex samples. an approach may involve synthetic data generation using smogn @xcite . @xcite acknowledge a reader's familiarity with a genre may affect perceived word complexity. however, the complex dataset lacks information on each annotator's expertise or background, which may offer valuable new insights.