CL ex IS 2: A New Corpus for Complex Word Identification Research in Computing Studies (2021)

Abstract: Reading is a complex process not only because of the words or sections that are difficult for the reader to understand. Complex word identification (CWI) is the task of detecting in the content of documents the words that are difficult or complex to understand by the people of a certain group. Annotated corpora for English learners are widely available, while they are less common for the Spanish language. In this article, we present CLexIS ^2 , a new corpus in Spanish to contribute to the advancement of research in the area of Lexical Simplification, specifically in the identification and prediction of complex words in computing studies. Several metrics used to evaluate the complexity of texts in Spanish were applied, such as LC, LDI, ILFW, SSR, SCI, ASL, CS. Furthermore, as a baseline of the primer, two experiments have been performed to predict the complexity of words: one using a supervised learning approach and the other using an unsupervised solution based on the frequency of words on a general corpus.

## introduction reading is a complex process not only because of the words or sections that are difficult for the reader to understand. therefore, an adequate understanding of the content of the texts is required to be able to create coherent mental representations and this way to be able to capture their content @xcite . information technologies make it possible for people to access abundant information in different areas such as education, news, social, health, or government, among others. however, this information is not accessible to many, since some people face great reading barriers such as long sentences, unusual words, or complex linguistic structures that do not allow them to understand the content of the texts, with people with intellectual disabilities and people being directly affected in learning; including university students, who are people with a high educational level and specialized knowledge in different subjects of study but, still, could be part of groups of people with reading disabilities @xcite . complex word identification (cwi) is the task of detecting words in the contents of texts that are difficult or complex for people in a certain group to understand @xcite . cwi and the substitution of words identified as complex may significantly improve readability and understandability of a given text @xcite . in recent years, cwi has become an area of great interest for the scientific community and researchers in computational linguistics proposing development of computational semantic analysis systems as evidenced by the shared tasks of cwi by @xcite in @xcite @xcite , the task of the cwi of the alexs 2020 contest, headquarters of iberlef 2020 by @xcite , and 15th edition of se-meval but the first lexical complexity prediction task. @xcite . annotated english learner corpus are widely available, spanish large learner corpus are far less common @xcite . although there are corpus for natural language processing (nlp) research in spanish, they do not contain the necessary annotations to develop reading comprehension tools for students in computing science. our aim is to begin to address the lack of data recorded in the corpora of written learner spanish. this article introduces the creation of a new corpus in spanish to contribute to the advances of research in the area of lexical simplification, specifically in the identification and prediction of complex words in computing studies. the corpus is named clexis 2 , and it is made up of a collection of academic texts from the degrees in computer systems engineering and the software degree of the faculty of mathematical sciences of the university of guayaquil (ecuador), a public institution, and one of the largest and oldest in the country, with around 67,000 students (according to the census in 2019). the complex word identification (cwi) shared task organized as part of the 13th workshop on innovative use of nlp for creating educational applications (bea), hosted in conjunction with naacl-hlt'2018, focused on multilingualism and provided data sets containing four languages: english, german, french, and spanish. according to @xcite the goal of the cwi task was to predict which words challenge non-native speakers based on annotations collected from native and non-native speakers. @xcite proposed the corpus of spanish learners (caes -acronym in spanish) (corpus of spanish learners in english). (segura-bedmar and martinez, 2017) used the easydpl (easy drug package leaflets) corpus, a collection of 306 booklets written in spanish and manually annotated with 1400 adverse drug effects and their simplest synonyms. the objective of this work was to improve the readability of leaflets by replacing the terms that describe the effects of drugs with simpler synonyms. they used a vector from a previously trained word embedding model. ## a new dataset the creation of clexis 2 gave rise to a new data set in the scope of academic courses at a higher education level. the process of preparing the texts is detailed below: as a first step, the subjects that make up each semester of study were identified. the first four semesters correspond to the software career and the following four semesters to the computer systems engineering career, giving a total of eight study semesters, where each semester consists of five subjects. next, the recordings of the classes (academic videos) taught by teachers in virtual classes in the last two semesters of study were selected, which were stored in each teacher's work cloud. using the dictation 1 application, the transcription process of each of the academic videos was carried out. the automatic transcriber did not have precision in the accentuation of the words as well as in the punctuation of the sentences, therefore, a process of grammar revision was carried out manually in each text to achieve its correct accentuation; it was also considered vitally important to separate the text into sentences for better understanding. finally, the number of texts per subject corresponds to an average of 100 texts, and in turn, each text contains an average of 77.29 words. ## lexical complexity of the corpus the evaluation of the complexity of the clexis 2 corpus texts was carried out by applying the seven measures of lexical complexity for the spanish language as in @xcite . these formulas were proposed by @xcite , with the exception of the ssr whose measurement was provided by @xcite . the detail is as follows: the lexical complexity index -lc. lexical distribution index -ldi. ## the experiments in cwi on the new corpus we carried out two experiments following two major approaches in cwi: based on the definitions of the experiments for the identification of difficult words in spanish carried out by @xcite and also those made by @xcite from the calculation of low frequency words, being those words whose frequency is less than 1,000, we built a system that allowed the detection of complex words contained in the clexis foot_0 corpus using crea 2 . in fig. 6 you can see the process flow implemented to automatically detect if a word is complex or not based on crea. next, the confusion matrix was performed to determine the effectiveness of the classification system used. we evaluated the effectiveness of this simple approach in terms of precision, recall, f1-score and accuracy. the results show that, the proportion of predictions that the model classified correctly corresponds to an accuracy of 0.4394, a precision of 0.5165 that belong to the really correct positive identifications, the hit rate allowed us to obtain a recall of 0.4709, and the proportion of real negatives correctly identified whose measure is the specificity with a value of 0.3963, finally, an f1 of 0.4927 was obtained that indicates the precision and robustness of the applied model. the analysis of the data determines that the software career has a higher lexical complexity than the computer systems career. once again, first-semester students find it difficult to move from high school to undergraduate education. the subjects that first semester students have are:programming algorithms and logic, differential calculus, democracy, introduction to software engineering, and language and communication. ## conclusions a new corpus was created and made available. we believe that it becomes a fundamental resource for the identification of complex words in computer science studies, which means a very useful resource for the development of effective nlp tools for university students. the texts used as a central source of linguistic information reveal the difficulties faced by students of computer science studies. the application of the lexical complexity metrics allowed evaluating the complexity of the content of the corpus texts, determining that in a large number of subjects, the lexicon that teachers use when teaching their classes contains complex sentences, a technical language and sophisticated causing difficulty in the understanding of students. future works could propose solutions that involve the creation of tools applying lexical simplification that greatly contribute to the contribution of students with low reading comprehension or intellectual disabilities to better understand the content of texts in the area of computer science. a possible solution will be the creation of a system that transforms complex texts into accessible ones, benefiting mainly university students in computer science who have disabilities and those who have reading comprehension difficulties. the best result obtained for the predictive value of the words for the data set was: mae of 0.060970, mse of 0.005889 and rmse of 0.09687 in the configuration with 240 nodes and 15 selected characteristics. the resource is available and can be shared by contacting the authors.