M anchester Metropolitan at S em E val-2021 Task 1: Convolutional Networks for Complex Word Identification (2021)

Abstract: We present two convolutional neural networks for predicting the complexity of words and phrases in context on a continuous scale. Both models utilize word and character embeddings alongside lexical features as inputs. Our system displays reasonable results with a Pearson correlation of 0.7754 on the task as a whole. We highlight the limitations of this method in properly assessing the context of the target text, and explore the effectiveness of both systems across a range of genres. Both models were submitted as part of LCP 2021, which focuses on the identification of complex words and phrases as a context dependent, regression based task.

## introduction complex word identification (cwi) involves identifying words that the reader may find difficult to understand. a word's complexity can depend on many factors and differ according to context. further, assessment of the complexity of named entities can require some degree of general knowledge, making cwi a challenging task @xcite . accurately identifying complex words is important for many downstream simplification tasks, making literature more accessible for people with conditions such as dyslexia @xcite , and the assessment of a text's readability as a whole @xcite . our methodology plans to extend on previous convolutional network based approaches to cwi @xcite . with the goal of producing a system that is able to better model the complexities of phrases and unfamiliar words, within the english language. previous shared tasks on cwi addressed the problem as a binary and probabilistic classification type task, although human judgements on word complexity are not binary and exist on a continuous scale. lexical complexity prediction (lcp) 2021 tries to address this and uses an augmented version of complex @xcite , a dataset annotated with a 5-point likert scale. complex also features context-specific annotation, with words receiving different annotations depending on their context. the dataset provides annotations from three different domains: bible, biomed and europarl @xcite . the code for this task is available on github foot_0 . ## related work word frequency is a commonly used feature in cwi @xcite ; words that appear frequently in language are more likely to be recognised and understood by the reader @xcite . for the purpose of identifying medical terminology that may be unfamiliar to the lay reader, @xcite leveraged lexical frequencies while also exploring the potential of other features such as word familiarity ratings from the mrc psycholinguistic database @xcite . more recently lexical and psycholinguistic features have been utilized by machine learning tools, resulting in improved accuracy on these tasks. through the use of an enseble-based voting method the camb system @xcite achieved state-of-the-art results in the 2018 cwi shared task @xcite , employing a total of 27 lexical, morphological and psycholinguistic features. the camb system however does not consider the target words context, opting for a "greedy" approach towards phrase classification, marking all phrases as complex. aroyehun et al. (2018) explored the use of convolutional neural networks (cnn) for cwi using only the word embeddings of the target words and the averaged embeddings of the left and right contexts as inputs. they contrasted the results against a feature engineering approach using decision tree learning finding that both methods achieved competitive results. however, their decision tree method was marginally more accurate than their cnn for most of the datasets. integrating lexical features alongside word embeddings can lead to further improvements in accuracy making this a more competitive approach, and outperforming many previous deep learning methods for cwi @xcite . by framing cwi as a sequence labelling task, bi-directional long short-term memory (bilstm) networks have produced state-of-the-art results on the cwig3g2 dataset @xcite . bilstm networks are able to capture long-term word and character level dependencies allowing these models to consider a large amount of contextual information. modelling the complexity of phrases has proven to be a more challenging and complex task compared to individual words @xcite . ## discussion in this paper, we presented two convolutional neural networks used as an approach to singleword and multi-word complex word identification. both models achieved reasonable results, achieving scores in a comparable range to the majority of other submissions. multi-word cwi is a more challenging task compared to the assessment of single words; the multiword model was able to utilize the datasets of both tasks, and its predictions show a pearson's correlation score of 0.7611. our system is only able to process two-word mwes, which for this task is not an issue. however, in other use cases the ability to assess longer mwes would be useful. given a dataset with annotations for longer mwes the model could potentially be adapted to work with three or four word sequences. both models are able to assess the context of the target text when making predictions; although, as the left and right contexts are given as an average, all words are weighted equally regardless of their relevance to the target text. because of this equal weighting of words, the models are able to adjust their predictions based on the general complexity of the sentence but are unable to fully capture the relevant context. adding a mechanism that could weight each word in the context based on certain features may offer some improvements in this area. attention based models such as bert @xcite are able to attend to each token in a sequence to produce embeddings that capture large amounts of contextual information. finetuning such a model on cwi tasks could produce embeddings that contain more useful and relevant information.