Interpreting Neural CWI Classifiers’ Weights as Vocabulary Size (2020)

Abstract: Complex Word Identification (CWI) is a task for the identification of words that are challenging for second-language learners to read. Even though the use of neural classifiers is now common in CWI, the interpretation of their parameters remains difficult. This paper analyzes neural CWI classifiers and shows that some of their parameters can be interpreted as vocabulary size. We present a novel formalization of vocabulary size measurement methods that are practiced in the applied linguistics field as a kind of neural classifier. We also contribute to building a novel dataset for validating vocabulary testing and readability via crowdsourcing.

## introduction the readability of second-language learners has attracted great interest in studies in the field of natural language processing (nlp) @xcite . as nlp mainly addresses automatic editing of texts, readability assessment studies in this field have focused on identifying complex parts by assuming that the words identified are eventually simplified so that learners can read them. to this end, complex word identification (cwi) @xcite tasks have been studied extensively. recently, a personalized cwi task has been proposed, where the goal of the task is to predict whether a word is complex for each learner in a personalized manner @xcite . neural models are also employed in these studies and have achieved excellent performance. the weights, or parameters, of a personalized high-performance neural cwi, obviously include information on how to measure the word difficulty and learner ability from a variety of features. if such information could be extracted from the model in a form that is easy to interpret, it would not only be use @xcite @xcite @xcite @xcite . to this end, this paper proposes a method for interpreting the weights of personalized neural cwi models. let us suppose that we have a corpus and that its word frequency ranking reflects its word difficulty. using our method, a word's difficulty can be interpreted as the frequency rank of the word in the corpus and a learner's ability can be interpreted as the vocabulary size with respect to the corpus, i.e., the number of words known to the learner when counted in a descending order of frequency in the corpus. our key idea is to compare cwi studies with vocabulary testing studies in applied linguistics @xcite . second-language vocabulary is extensive and occupies most of the time spent in learning a language. vocabulary testing studies focus on measuring each learner's second language vocabulary quickly. one of the major findings of these studies is that a learner needs to "know" at least from 95% to 98% of the tokens in a target text to read. here, to measure if a learner "knows" a word, vocabulary testing studies use the learner's vocabulary size and word frequency ranking of a balanced corpus. hence, by formalizing the measurement method used in vocabulary testing studies as a neural personalized cwi, we can interpret neural personalized cwi models' weights as vocabulary size and word frequency ranking. our contributions are summarized as follows: ## proposed formulation the following notations are used. we have j learners {l 1 , l 2 , . . . , l j , . . . , l j } and i words {v 1 , v 2 , . . . , v i , . . . , v i }. j is the index of the learners and i is the index of the words. when there is no ambiguity, we denote word v i as word i and learner l j as learner j, for the sake of simplicity. let k i be the number of occurrences of word v i . while we do not use in our experiments, for generality, we explicitly write the index for each of the occurrences, i.e., k. let u i,k be the k-th occurrence of word v i in the text. let b j be the ability of learner j and let d i,k be the difficulty of the k-th occurrence of word v i . a dichotomous decision using a neural networkbased formulation is typically modeled using a probabilistic formulation. let y j,i,k be a binary random variable that takes 1 if learner l j knows the k-th occurrence of word v i , otherwise it takes 0. subsequently, it is typical to use a function that maps a real number to the [0, 1] range so that the real number can be interpreted as a probability. to this end, typically σ is the logistic sigmoid function, i.e., σ(x) = 1 exp(-x)+1 is used. then, the probability that learner l j knows the k-th occurrence of word v i , namely, u i,k , can be modeled as in eq. 2. probability in a dichotomous classification is most ambiguous when it takes 0.5. by focusing on the point the vertical line takes 0.5, we can see that the sign of b j -d i,k determines whether or not the probability is larger than 0.5. ## conclusions in this paper, we theoretically showed that previous vocabulary size-based classifiers can be seen as a special case of a neural classifier. we also built a dataset necessary for this evaluation and made it publicly available in the form of an attached dataset. future work include more detailed experiments on language learners' second language vocabularies.