J apanese Lexical Complexity for Non-Native Readers: A New Dataset (2023)

Abstract: Lexical complexity prediction (LCP) is the task of predicting the complexity of words in a text on a continuous scale. It plays a vital role in simplifying or annotating complex words to assist readers. To study lexical complexity in Japanese, we construct the first Japanese LCP dataset. Our dataset provides separate complexity scores for Chinese/Korean annotators and others to address the readers’ L1-specific needs. In the baseline experiment, we demonstrate the effectiveness of a BERT-based system for Japanese LCP.

## introduction reading comprehension requires a certain level of vocabulary knowledge. the results reported by @xcite suggest that most english learners need to understand 98% of tokens in a text to comprehend it. a follow-up study by @xcite estimates the percentage to be 96% for japanese learners to comprehend text. acquiring vocabulary to reach such levels, in turn, is a lengthy and challenging task for learners. this opens up opportunities for assistive applications, such as simplification or annotation of complex words. the first step necessary for such applications is to predict the complexity of the words. the task of lexical complexity prediction (lcp) is defined as predicting how difficult to comprehend words or phrases in a text are on a continuous scale @xcite . this differentiates lcp from complex word identification (cwi), i.e., binary classification of complex words @xcite . as complexity is naturally perceived as continuous, a continuous scale used in lcp allows to represent it without loss of information. the lcp research so far has been limited to english, for which two lcp datasets have been constructed @xcite , and no such dataset has been created for japanese. meanwhile, there are a number of features specific to the japanese language that could affect lexical complexity, and their effects have yet to be studied. for example, the chinese characters, which are used extensively in japanese, lower text readability @xcite . previous studies on japanese lexical complexity used pedagogical word lists to estimate complexity level. nishihara and kajiwara (2020) modeled lexical complexity of words based on the japanese educational vocabulary list @xcite . the word list assigns a degree of difficulty to each item, based on the subjective judgment of japanese language teachers, not learners themselves, and does not consider the learners' l1 background. in light of this, we present jalecon foot_0 , dataset of japanese lexical complexity for non-native readers. our dataset has the following key features: our analysis reveals that the non-chinese/korean annotators perceive words of chinese origin or containing chinese characters as especially complex. in the baseline experiment, we investigate the effectiveness of a bert-based system in the japanese lcp task, and how it varies according to the word complexity and l1 background. ## task setting since japanese has no explicit word boundaries, word segmentation is the first prerequisite for lcp. we use short unit words (suws) as the basic word unit, combining them into longer word units in the case of multi-word expressions (mwes): suws 右肩 上がり に 増え て いる right.shoulder rise adv increase ger be-prs words 右肩上がり mwe steady.rise に suw adv 増え suw increase ている mwe prg-prs "is steadily increasing" figure 1: example of text segmented as suws and as words (either suw or mwe ). semantically opaque sequences are chunked into mwes. abbreviations in glosses: adverbializer, gerund, present, progressive. suw: suws consist of one or two smallest lexical units @xcite , and are commonly used for segmentation of japanese. mwe: we understand mwes as multi-suw expressions that are fixed or semantically opaque (see appendix c) and consequently may have higher complexity than their components. we identify mwes either using long unit word (luw) foot_1 segmentation, or manually (see section 3). consequently, a word, can be either an suw or an mwe (see figure 1 for examples). a complexity score represents perceived complexity based on the annotators' judgment on a scale from 0 (least complex) to 1 (most complex). we exclude proper nouns from our target because their complexity is influenced by factors unrelated to reading proficiency or vocabulary knowledge. 3 we annotate the words in an in-context dense setting. in-context here means including both intrasentence and extra-sentence context of each word. context is important for lexical complexity for two reasons @xcite : (1) as polysemous words can have different complexity levels for each sense, context is necessary to differentiate between possible meanings of these words. (2) presenting a word without context could increase its complexity. in particular, the recognition of abstract words relies on context @xcite . dense means annotating each word of the text with a complexity label, instead of annotating one specific word in each sentence @xcite . we adopt the dense setting to avoid any bias that could arise from targeting specific words. ## construction of jalecon in order to include both written and spoken language and a variety of vocabulary, we sourced texts from two different genres: news comes from the japanese-english data of the wmt22 general machine translation task @xcite . it contains a variety of news texts written for the general japanese reader. government is composed of press conference transcripts from japanese ministries or agencies. 4 the whole dataset is composed of sequences of sentences constituting either the beginning of an article (news) or a question-answer pair (government). we restricted the length of the sequences to at least 6 and at most 11 sentences to obtain similar amounts of text, and presented each sequence as a whole for annotation. ## statistics and analysis overall statistics for both genres and l1 groups are shown in we measured inter-annotator agreement (iaa) using krippendorf's α for interval values @xcite . the iaa is 0.32 in the ck group, and 0.31 in the non-ck group, while it would be 0.19 if we merged the groups. as lexical complexity is japanese chinese other all cc all cc all cc ck .003 .009 .004 .004 .071 .000 non-ck .010 .032 .062 .072 .007 .143 frequency 52% 10% 26% 22% 4% 0% in addition to the aforementioned difference in proficiency, there is also a clear difference in how the two l1 groups perceive complexity of words based on their origin and whether they contain chinese characters 10 , as analyzed in ## experiments the newly created dataset can be used to evaluate performance of lcp for non-native japanese readers of different l1 backgrounds (ck and non-ck). we developed a baseline system based on a fine-tuned bert @xcite model, and evaluated it using cross-validation. we fine-tuned a japanese pre-trained bert model released by tohoku university, namely the base model for unidic lite segmentation 12 . for each word w in our dataset and the sentence s that contains it at token indices i to j -1, we construct an input sequence ([cls], we fine-tune and evaluate models for ck and non-ck complexity separately. see appendix g for the hyperparameters and crossvalidation scheme. the results are reported in ## conclusion in this paper, we presented the first dataset for japanese lcp. it provides separate complexity scores based on the ck/non-ck distinction of annotators' l1 background. our analysis corroborates our conjecture that special consideration of l1 background is useful for the japanese lcp task in particular. we believe it could benefit lcp in other languages as well. in the baseline experiment, we demonstrated the efficacy of our bert-based system for both ck and non-ck readers. even after separating ck and non-ck annotators, however, notable inter-annotator disagreement remains within these groups. therefore personalized systems analogous to @xcite could improve on our system. future research should study this possibility, analyzing both its costs and benefits. models trained on jalecon can be used as part of a lexical simplification pipeline for japanese, both to identify complex words and to rank candidate simplifications. jalecon itself can be further used as a basis for a lexical simplification dataset targeting words actually perceived as complex, similar to tsar-st datasets for english and spanish @xcite .