Text Retrieval for Language Learners: Graded Vocabulary vs. Open Learner Model (2021)

Abstract: A text retrieval system for language learning returns reading materials at the appropriate difficulty level for the user. The system typically maintains a learner model on the user’s vocabulary knowledge, and identifies texts that best fit the model. As the user’s language proficiency increases, model updates are necessary to retrieve texts with the corresponding lexical complexity. We investigate an open learner model that allows user modification of its content, and evaluate its effectiveness with respect to the amount of user update effort. We compare this model with the graded approach, in which the system returns texts at the optimal grade. When the user makes at least half of the expected updates to the open learner model, simulation results show that it outperforms the graded approach in retrieving texts that fit user preference for new-word density.

## introduction since language learning requires extensive extracurricular reading, learners can benefit from a text retrieval system that helps them identify suitable reading materials from a pool of candidate texts @xcite @xcite . the suitability of a text may depend on multiple factors, including the user's reading interests @xcite , and the degree of matching between its difficulty and the user's proficiency. this paper investigates the use of an open learner model (olm) to predict the latter. by giving users more control over their learning, olms have been shown to foster users' confidence and reflection on their progress @xcite . these benefits are especially important for long-term or life-long learning activities @xcite , such as foreign language learning. we evaluate the effectiveness of an editable olm @xcite -i.e., an olm that allows the user not only to view but also to modify its content -for text retrieval for language learning. as their vocabulary expands, users can update the olm so that the system continues to retrieve texts that are lexically challenging to them. specifically, we address two research questions: text retrieval performance (q1): how accurately can an olm identify reading materials with the desired density of new vocabulary, as specified by the user? user update effort (q2): how frequently does the user need to edit the olm in order to reap its benefits? most previous research in computer-assisted language learning measured text retrieval performance through holistic evaluation @xcite , or in terms of users' overall learning outcomes @xcite . in answering q1, we will directly evaluate the density of new vocabulary in the retrieved texts. in addressing q2, we will further consider how retrieval performance is affected by users' vocabulary acquisition over time and the amount of user update effort. the rest of the paper is organized as follows. after a review of previous work (section 2), we define our text retrieval framework (section 3). we then describe and motivate the user simulation (section 4). next, we present the olm approach and the baseline graded approach (section 5). finally, we compare the performance of the olm and the graded approach with respect to the amount of user update effort (section 6). ## previous work while text difficulty can be influenced by a variety of lexical, syntactic, semantic and discourse features, many text recommendation systems focus on vocabulary @xcite @xcite , likely due to the strong correlation between vocabulary difficulty and text difficulty @xcite . our study will similarly adopt vocabulary difficulty as the text retrieval criterion. various approaches have been proposed for matching language learners with reading materials at a suitable level of lexical difficulty. the graded approach, also known as the leveling approach, places users and documents on a common scale, such as school grades @xcite . the system performs automatic readability assessment on each document and labels it with a grade to reflect its difficulty. this approach may not be able to capture individual learning patterns, however, as users are pigeon-holed into pre-defined grades. as an alternative, the adaptive approach identifies user traits and preferences, and then adjusts the pedagogical content in the system to optimize learning outcomes @xcite . in the context of text retrieval for language learning, the system typically maintains a learner model on the user's linguistic proficiency, and then returns texts that best fit the model. the learner model may be estimated through user updates (lee, 2021); formal assessment such as cloze items @xcite ; complex word identification models trained on vocabulary selfassessment by the user (yeung and lee, 2018); time log and click history patterns @xcite ; as well as dictionary and translation queries by the user @xcite , among other non-invasive methods. one disadvantage of the graded approach is the "jump" in difficulty when promoting the user from one grade to the next. the adaptive approach can potentially provide more fine-grained adjustments by gradually raising the vocabulary difficulty in the retrieved texts. to the best of our knowledge, there has not been any direct, quantitative comparison between these two approaches during a period of vocabulary acquisition by the user. this paper aims to fill in this gap. ## text retrieval framework after motivating the use of vocabulary difficulty as the retrieval criterion (section 3.1), we describe its implementation in the learner model (section 3.2) and its application in the retrieval model (section 3.3). ## methodology after motivating the advantages of using a simulation to compare the open learner model (olm) and the graded vocabulary approach (section 4.1), we give details on the simulation set-up (section 4.2) and implementation details (section 4.3) and define the evaluation metrics (section 4.4). ## approach as users learn new words, a document's new-word density (nwd) decreases. periodic updates to the vocabulary set and re-estimation of the nwd of candidate documents are therefore necessary to ensure that the retrieved texts remain adequately challenging. we compare two approaches for this task. ## results we conducted the simulation with a database of 1923k chinese wikipedia entries and 29k short essays. 1 we performed automatic word segmentation on all documents with the stanford corenlp parser @xcite . we now present a chronological analysis of the simulation (section 6.1), and then examine the overall experimental results (section 6.2). ## conclusions automatic text retrieval supports language learners in self-directed reading and independent learning. a major challenge in this task is to match learners with different capabilities to texts with appropriate vocabulary complexity. we have evaluated an open learner model (olm) that allows users to update their individual progress in vocabulary acquisition. we compared this model to the graded approach, where the system recommends texts to users at the optimal grade. we conducted a simulation of a learner of chinese as a foreign language who uses the retrieval system during a period of vocabulary acquisition. results show that the olm outperforms the graded approach in retrieving texts at a range of target nwds. when the user makes at least half of the expected updates, the olm's fine-grained, incremental adjustment yields superior retrieval performance. we believe these results can help inform the design of text retrieval systems for language learners. in future work, we intend to further improve retrieval quality by extending the olm beyond vocabulary to other dimensions of text difficulty, such as syntactic and semantic complexity.