Complex Word Identification for I talian Language: A Dictionary–based Approach (2024)

Abstract: Assessing word complexity in Italian poses significant challenges, particularly due to the absence of a standardized dataset. This study introduces the first automatic model designed to identify word complexity for native Italian speakers. A dictionary of simple and complex words was constructed, and various configurations of linguistic features were explored to find the best statistical classifier based on Random Forest algorithm. Considering the probabilities of a word to belong to a class, a comparison between the models’ predictions and human assessments derived from a dataset annotated for complexity perception was made. Finally, the degree of accord between the model predictions and the human inter-annotator agreement was analyzed using Spearman correlation. Our findings indicate that a model incorporating both linguistic features and word embeddings performed better than other simpler models, also showing a value of correlation with the human judgements similar to the inter-annotator agreement. This study demonstrates the feasibility of an automatic system for detecting complexity in the Italian language with good performances and comparable effectiveness to humans in this subjective task.

## introduction identifying the complexity of a word is a very challenging process that requires a series of linguistic reflections intertwined with the concept of complexity itself @xcite . while humans can intuitively perceive word simplicity, translating this intuition into quantitative parameters for automatic systems is challenging. the task of complex word identification (cwi) aims to pinpoint those words that may pose decoding challenges for certain readers due to a variety of linguistic features @xcite . the concept of linguistic complexity indeed is closely intertwined with the readability and accessibility of texts @xcite . recognizing complex words is crucial, not only for readers with learning difficulties, such as dyslexia or aphasia @xcite , but also for native speakers, since understanding word meanings is fundamental for comprehension @xcite . studies related to cwi have seen a significant increase in recent years, either as a part of lexical simplification systems @xcite , or as an independent task, promoted by several shared tasks @xcite @xcite . in the latter case, it is very useful for the development of systems aiming at facilitating foreign language acquisition, creating reading tools for individuals with limited linguistic skills, and enhancing accessibility for native speakers @xcite . despite the importance of cwi, the development of such systems has been limited to a few languages, mainly due to the scarcity of necessary linguistic resources and the high costs associated with their development @xcite . to the best of our knowledge, there have been no studies directly addressing the cwi in the italian language, even though research has focused on text simplification @xcite . the absence of requisite databases classifies italian as a 'low-resource language' for this specific task. the main contribution of this article is to propose the first automatic system to identify lexical complexity specifically designed for native italian speakers, motivated by educational concerns @xcite and the need to understand perceived complexity under typical conditions. we created a dataset of individual lexical entries, labelled as simple or complex (3.1) and selected various linguistic features (3.2), through which a classifier system could be trained in a supervised setting (3.3). our approach is dictionary-based and context agnostic @xcite . we considered the probability of each item belonging to a certain class as the prediction of word complexity. finally, the system was validated against a dataset containing human judgements regarding the perceived complexity of selected words (3.3). ## related work recent investigation in cwi focused on the development of statistical classifiers that can accurately assign lexical items to specific complexity classes based on labelled data @xcite . classification systems typically utilize feature-based approaches or neural networks with word embeddings to enhance prediction accuracy @xcite . most cwi studies classify word complexity in two primary ways: binary classification, labeling words as either simple or complex (0-1), and continuous classification, where words receive a complexity score on a continuum from very simple to very complex. in recent years, it has become more common to use lexical complexity prediction name to refer to the latter @xcite . among the statistical classifiers, support vector machines, decision trees, random forests, logistic regression, and recurrent neural networks have been prominently used @xcite . for the italian language, the few studies concerning lexical simplification @xcite have overlooked this task deemed crucial for the proper execution of simplification @xcite . the words to be simplified were selected exclusively on the basis of the frequency parameter @xcite relying on nuovo vocabolario di base @xcite , which is a fundamental lexicon for the italian language, comprising approximately 7,000 selected words. this approach poses significant limitations, as words outside this vocabulary are often prematurely considered complex, and potential substitutions are restricted to those within the same lexicon. this approach does not take into account the nuanced and multifaceted nature of linguistic complexity, so relying on a single measure such as frequency can lead to oversimplification @xcite : frequency is strongly linked to the reference corpus used to calculate it. ## methods in this study, we developed a binary classification system using a dataset of isolated words, created due to the absence of comprehensive resources for the cwi task. recognizing that word complexity is intrinsically context-dependent and that complexity itself is a gradient, our approach was constrained by resource limitations. the creation of a large-scale dataset capable of training models with nuanced human judgments in context would require significant time and resources. consequently, we opted for a more manageable solution by employing a word list, which is computationally 'small and easily tractable' @xcite . this choice was motivated by its ability to provide representative data for our purpose. the decision to adopt binary classification reflects not only these practical constraints but also avoids the subjectivity inherent in gradual classifications without extensive contextual data. this pragmatic approach aims to establish a foundational methodology that can be expanded as more comprehensive data become available. we selected various linguistic features to characterize the complexity of our items. from these features, our model learned to predict the complexity classification and the likelihood of a word belonging to a particular class. while it is recognized that the probability of a target word being classified as simple or complex does not directly predict its degree of complexity @xcite , the continuous probabilistic values generated by our model provide valuable insights into the nuanced nature of word complexity. these probabilistic values reflect the uncertainty inherent in the classification process: higher probabilities indicate a stronger likelihood that a word has intricate linguistic properties, whereas lower probabilities suggest simpler linguistic structures. these predicted values can then be compared with complexity assessments derived from human judgments, which serve as our gold standard. this methodology helps bridge the gap between objective classification and subjective perception, enhancing our understanding of lexical complexity. ## conclusions in this study, we introduced the first system aimed at identifying complex words within the italian language, marking the initial exploration of this task for this linguistic domain. the absence of specific datasets prompted us to build a dictionary comprising approximately 13,000 words annotated for simplicity and complexity. an appropriate selection of descriptive features of word complexity made it possible to train a classification model in different configurations. we tested our models on a test set and on an external dataset, containing human judgements on word complexity, our gold standard. from the different validation analyses we saw that the best model is total model that integrates the linguistic features with the word embedding. we conducted a further analysis by comparing the total model to our gold standard. this dataset was annotated by multiple native italian speakers and for this reason we decided to calculate the inter-annotation agreement and compared it with model-human correlation. in this way we not only validated the reliability of our dataset and the fidelity of our predictive model but also established the basis for a meaningful comparison between human and machine assessments. our analysis revealed that the average correlation of each predicted value from our model with the inter-annotator agreement falls within the same range, suggesting that our system is as effective as human judgment in subjective tasks such as this. to enhance inter-annotator agreement and the robustness of our findings, future efforts will focus on increasing the sample size and the number of annotators. expanding the sample size will cover a broader lexical domain and provide a diverse set of words and contexts, thereby improving the model's generalizability to unseen data. this broader coverage supports robust statistical testing and validation, minimizing the influence of outliers. incorporating more annotators is crucial for enriching the diversity of perspectives in the evaluation process, which is particularly important in subjective assessments where personal experiences, linguistic backgrounds, and individual biases might skew judgments. a larger pool of annotators diminishes these biases, fostering a balanced and representative consensus on lexical complexity. furthermore, this approach allows for more detailed inter-annotator agreement analyses, clearly highlighting areas of consensus and disagreement. together, these strategies not only enhance the reliability of our annotations but also improve the overall accuracy and applicability of our model. the main limitation of our approach resides in the characteristics of the dataset we used to train our models. our dataset is composed by words presented in isolation, thus disregarding crucial proceedings of clib 2024 contextual cues essential for understanding word meanings and disambiguation. we acknowledge the critical role of context in complexity analysis and recognize the necessity of incorporating specific contextual information where the target word appears. moving forward, our aim is to advance in this direction by expanding effective datasets that integrate contextual frameworks for training our word cwi systems. although challenges persist in the field of cwi, our study lays some groundwork for exploring this task for italian and underscores the potential of automated systems in this domain. in the future, collaborative efforts and advancements in building datasets and refining models will be crucial for advancing the field and uncovering new insights into language complexity. this methodology could enhance the precision of readability measures @xcite , particularly in terms of lexical range and lexical sophistication. moreover, such a system can be an essential component in a text simplification pipeline. by identifying words that may pose comprehension challenges, the system not only flags these words for potential replacement but also assists in suggesting simpler alternatives. this functionality ensures that the replacements not only match the original words' meanings as closely as possible but also contribute to a text that is overall easier to understand. while this study focuses on the italian language, the methodologies and models we have developed have the potential to be adapted for other languages, especially those considered low-resource in the context of computational linguistic tools. by leveraging similar linguistic resources and adjusting the feature sets to accommodate languagespecific characteristics, researchers can extend this approach to support complex word identification across diverse linguistic domains.