To What Extent Does Lexical Normalization Help E nglish-as-a-Second Language Learners to Read Noisy E nglish Texts? (2021)

Abstract: How difficult is it for English-as-a-second language (ESL) learners to read noisy English texts? Do ESL learners need lexical normalization to read noisy English texts? These questions may also affect community formation on social networking sites where differences can be attributed to ESL learners and native English speakers. However, few studies have addressed these questions. To this end, we built highly accurate readability assessors to evaluate the readability of texts for ESL learners. We then applied these assessors to noisy English texts to further assess the readability of the texts. The experimental results showed that although intermediate-level ESL learners can read most noisy English texts in the first place, lexical normalization significantly improves the readability of noisy English texts for ESL learners.

## introduction noisy english texts are not only problematic for processing text but also difficult for humans to read, even for native speakers when, for instance, they lack the background to decipher abbreviations. how difficult are noisy english texts for english as a second language (esl) learners? if noisy english texts are too difficult for esl learners to read, it may affect their behavior on various media such as social networking services (sns). for example, it is easy to imagine that esl learners do not follow english accounts. the lexical normalization tasks are important as they could potentially change the behavior of esl learners by improving readability. in this sense, assessing the readability of noisy english texts for esl learners is closely related to assessing the extent to which the language gap leads to social division. to this end, we first sought to understand the readability of noisy english texts by building highly accurate readability assessors. we used two approaches to build the assessors. the first was based on the field of educational natural language processing (nlp) @xcite . using a corpus that is standard in this field, we built a highly accurate readability assessor using deep learning methods, such as bidirectional encoder representations from transformers (bert) @xcite . the second approach was to conduct readability assessments based on the vocabulary of english learners. these methods have been well studied in applied linguistics, whereby considerable research has revealed that english learners need to know more than 95% of the words in a text to read and understand them @xcite . the idea of assessing text readability via each learner's vocabulary knowledge is beneficial for interpreting the readability assessment results. therefore, we also constructed a classifier based on a dataset of english learners' vocabulary tests @xcite , to determine the number of words in a text that an english learner knows. in experiments carried out on a standard dataset for evaluating readability @xcite in educational nlp, the results provided by the two approaches were in close agreement. the experiments using an english noisy text corpus also showed that most noisy english texts were readable by intermediate english learners. using goldnormalized texts in the noisy english text dataset, we found that lexical normalization improves the text readability of second language learners. this improvement in readability was statistically significant in the vocabulary-based assessor's results. the contributions of this study are as follows: ## automatic readability assessment this section formalizes the problem of automatic readability assessment. let us suppose that we have n texts to assess: we write the set of texts as {t i |@xmath0, . . . , n }}. let y be the set of readability labels. labels are typically ordered in the order of difficulty. for example, in the on-estopenglish dataset @xcite , we can set @xmath1, 1, 2}, where 0 is elementary, 1 is intermediate, and 2 is advanced. the number of levels depends on the evaluation corpus. using y, we write the label for t i as y @xmath2. given each text t i , an assessor outputs its readability score s i . in a supervised setting, the assessor knows the number of levels in the evaluation corpus from training examples. hence, s i ranges within y: s @xmath3. however, in an unsupervised setting, it is noteworthy that the assessor does not know y, or how many levels the evaluation corpus has, because no label is given. hence, even if only integers are allowed for y i , s i can be a real value. throughout this paper, we write arrays using @xcite . given n texts [t i |@xmath4, . . . , n }], our goal is to make an assessor output arrays of readability scores [s i |@xmath5, . . . , n }] that correlate well with the array of labels [y i |@xmath6, . . . , n }]. here, there are multiple types of correlation coefficients between the array of scores and the array of labels, which we explain in the later sections. typically, we should use rank coefficients such as spearman's ρ, defined as the pearson's ρ between rankings, when s i is real-valued. ## vocabulary testing-based readability fig. 1 shows example questions from the vocabulary size test, a widely used vocabulary test in applied linguistics @xcite . each question asks about a word in a multiple-choice question format. the test consists of 100 questions like those shown in fig. 1 . @xcite used this test to have 100 second-language learners take the test and to collect their responses. their data were published and made publicly available. we used 15. deficit: the @xmath7. a: spent a lot more money than it earned b: went down a lot in value c: had a plan for its spending that used a lot of money d: had a lot of money stored in the bank @xcite , one of the most widely accepted vocabulary tests to quickly assess language learners. they are asked to choose the option that paraphrases the part between "<" and ">" from a, b, c, and d. we want to analyze vocabulary test results to obtain word difficulty values encoding learners' language knowledge. to this end, we employed the idea of item response theory @xcite , a statistical model that can estimate learners' abilities and test questions' difficulties from the learners' responses to the questions. let v be the set of vocabulary, and let l be the set of learners. let z v,@xmath8, 1} be the result of whether learner @xmath9: z l,@xmath10; otherwise, z l,@xmath11. correct answers usually imply that l knows word v. then, by using {z v,l } as the training data, we train the following model: in eq. 1, a l is the ability parameter of learner l, d v is the difficulty of word w, and sigmoid denotes the logistic sigmoid function, i.e., sigmoid(x) = 1 1+exp(-x) . the logistic sigmoid function is the binary version of the softmax function, which is frequently used in neural classifiers. it is a monotonously increasing function ranging within (0, 1). as sigmoid(0) = 1 1+1 = 1 2 , when a learner's ability a l is larger than the word difficulty d v , the probability that learner l knows word v can be written as follows: p(@xmath12, l) > 1 2 in eq. 1. likewise, by using eq. 1, we can compare a learner's ability and word difficulty in the same dimension. to estimate learner ability and word difficulty, z v,l is given as z in eq. 1 in the training phase. in this way, in item response theory, learner ability and word difficulty are comparable, and these parameters are estimated from the test result data. in eq. 1, d v denotes the word difficulty estimated from the vocabulary tests. here, in addition to the word difficulty for the words within the vocabulary test, we also want to obtain word difficulty values for all words that may appear in the target language. to this end, we calculate d v from the word frequency in large balanced corpora as follows: in eq. 2, k is the number of corpora to use, freq k (v) denotes the frequency of word v in the k-th corpus, and w k is the weight parameter of the k-th corpus. in summary, given the vocabulary test results {z v,l } and corpus frequency features freq k (v), we can estimate the parameters: namely, the weight of the k-th corpus w k and learner l's ability a l . to implement the model, we used logistic regression, by following @xcite . note that this model does not use the valuable readability label {y i } in the training phase, so is unsupervised. as shown in eq. 1, we employed irt-based modeling in this study. irt-based modeling has been used in many previous nlp studies such as @xcite @xcite @xcite . after estimating the parameters using the abovementioned procedure, we use the following formula to obtain the readability of given t i . here, l avg denotes the test-taker whose estimated ability parameter is closest to the average of the estimated ability parameter values {a l }s. intuitively, the following equation calculates the probability that the average learner knows all the words that appear in t i and uses it as the readability score. ## onestopenglish experiments we used the onestopenglish dataset @xcite for the source of readability for second language learners because it is one of the newest, publicly available, and reliable in the sense that no known trivial features are effective for predicting its labels such as average sentence length. the dataset has three levels: elementary, intermediate, and advanced. the original articles were taken from the guardian newspaper. the onestopenglish dataset is a parallel corpus, i.e, language teachers manually rewrote the original articles into the three aforementioned readability levels: hence, its readability labels are not easily estimated from the topics of texts. all three levels have 189 texts each, 567 texts in total. we randomly split these texts into a training set consisting of 339 texts, a validation set consisting of 114 texts, and a test set consisting of 114 texts. the training and validation sets were used to train solely supervised methods for comparison. unsupervised methods did not use the training and validation sets; they used only the test set. ## experiments with noisy english texts tab. 2 shows the readability assessment results using spvbert before/after lexical normalization. as most texts are elementary or intermediate, this result implies that intermediate-level esl learners can read most noisy english texts. we can also see that the difficulty of elementary texts can increase based on lexical normalization. tab. 3 displays the confusion matrix before/after lexical normalization by spvbert. we observe that lexical normalization between elementary and intermediate levels can work both ways: while 117 texts were converted from elementary to intermediate, 97 texts were converted from intermediate to elementary. no text was converted to the advanced level. while tab. 2 and tab. 3 show that lexical normalization improves readability, the improvement was not statistically significant because 97 texts were reversely converted. next, we explain the readability assessment results of vocabulary-based, which uses eq. 3. the smaller the value, the easier it is for esl learners to read. while the average score before lexical normalization was 4.00, the value after lexical normalization was 3.53. fig. 2 is the scatter plot of the readability scores before/after lexical normalization. fig. 2 clearly shows that the texts become significantly easier to read after lexical normalization (wilcoxon test, @xmath13.01). ## discussion in this study, we assessed the readability of text for esl learners utilizing two sources of information. spvbert is based on the annotations of the language teachers, and vocabulary-based is based on the results of the vocabulary test results of the learners. in this paper, both information from language teachers as well as learners is incorporated into the readability assessments described in the previous section. however, there are limitations to automated readability assessment, and ideally a large-scale survey of esl learners is required, which would be very costly and hence will be the subject of future work. even in manual assessment, there are points to consider. for example, in annotating the readability of texts by language teachers, there is the issue of how well the language teachers grasp the characteristics of the students. in addition, even if learners themselves participate in assessing text readability, the cultural background of the learner, such as the influence of their first language (l1), may influence the performance. ## conclusions we assess the readability of noisy english texts for esl learners. we built highly accurate assessors using the two approaches. while intermediate esl learners can read noisy english texts, lexical normalization improves readability for esl learners, as was the case with our dataset. future work will include more comprehensive experiments using other datasets and evaluations by actual esl learners.