Automatically Suggesting Diverse Example Sentences for L 2 J apanese Learners Using Pre-Trained Language Models (2024)

Abstract: Providing example sentences that are diverse and aligned with learners’ proficiency levels is essential for fostering effective language acquisition.This study examines the use of Pre-trained Language Models (PLMs) to produce example sentences targeting L2 Japanese learners.We utilize PLMs in two ways: as quality scoring components in a retrieval system that draws from a newly curated corpus of Japanese sentences, and as direct sentence generators using zero-shot learning.We evaluate the quality of sentences by considering multiple aspects such as difficulty, diversity, and naturalness, with a panel of raters consisting of learners of Japanese, native speakers – and GPT-4.Our findings suggest that there is inherent disagreement among participants on the ratings of sentence qualities, except for difficulty. Despite that, the retrieval approach was preferred by all evaluators, especially for beginner and advanced target proficiency, while the generative approaches received lower scores on average.Even so, our experiments highlight the potential for using PLMs to enhance the adaptability of sentence suggestion systems and therefore improve the language learning journey.

## introduction the term second language acquisition (or l2 acquisition) refers to the process of learning a second language by those who already know a first one. while children have a natural predisposition for acquiring languages, the degree of success among l2 learners varies greatly, as it is usually harder in adult life, requiring a combination of conscious effort, motivation, support from teachers and adequate materials @xcite . online dictionaries are usually the first resource towards which learners turn to in order to understand an unknown word or expression via definitions and example sentences. however, producing * research conducted during internship at nii, japan. ## related work in the following we discuss the related work, namely retrieving and generating example sentences, and estimating sentence difficulty. example selection similarly to @xcite , we seek to provide high-quality and diverse example japanese sentences. they propose a thorough retrieval approach based on quality and diversity scoring using a determinantal point process, and carry out an evaluation with l2 learn-1 more details on the jlpt website and section 4.1.1. ers and a teacher. our work differs from theirs in that we focus on selecting sentences for sense similarity given a target word in context, instead of many possible senses for a word in isolation. furthermore, we evaluate more aspects of the systems, in particular their capacity to adapt to learner proficiency levels. we also employ a language model in the evaluation. many other works deal with the task of example sentence selection from a corpus, focusing on dictionary examples for english, japanese and swedish @xcite @xcite . additionally, @xcite , @xcite and @xcite leverage parallel corpora to extract disambiguated sentences, while we limit our experiments to the monolingual setting. example generation there is a lot of research on controllable text generation approaches @xcite . possible generation targets are definitions for a given term @xcite , as well as example sentences. when it comes to example generation, researchers have shown that generated sentences can improve performance in word sense disambiguation tasks in a supervised @xcite or unsupervised way (he and yiu, 2022). focusing on l2 learners, @xcite consider lexical complexity and sentence length to generate example sentences of controllable difficulty. in our case, we opt not to rely on fixed sense inventories, primarily due to the scarcity of available sense-tagged corpora. however, we believe that assigning dictionary definitions to words could prove beneficial to learners. ## task: example sentence suggestion we define the l2 contextualized example suggestion task as: given a target word w, a context sentence s 0 and a target difficulty level d, we want to obtain a list of k good example sentences from a model m . to expand more on what makes a good example, @xcite suggest that such examples should represent typical usage, be informative and understandable to learners. building upon the discussion presented by @xcite , we aim to obtain multiple examples with diverse syntactic patterns since learners preferred them. ## evaluation 6.1 goals of the evaluation we outline the core research questions that guide our investigation. q1: the capabilities of llms such as gpt-4 in rating text have been explored @xcite . therefore, can gpt-4 evaluate the quality of japanese sentences from the perspective of l2 learners, and how do its assessments compare to those given by humans? q2: how do the automated quality metrics we used to guide the development of the retrieval approach compare with human judgment? q3: how good are plms at following instructions for this complex task? q4: is text retrieved from a corpus (assumed to be human-authored) preferred to generated text? q5: what do humans think of their output? we try to answer those questions by asking volunteer l2 learners and japanese native speakers to manually rate and rank systems outputs. ## results and discussion in this section, we present the results from the evaluation. we try to address our research questions in three main parts: agreement between raters; systems comparison; comments and error analysis. ## conclusion this paper outlines a methodology for suggesting example sentences to learners of japanese. it is adaptable to other languages with minor adjustments. the baselines we consider highlight many possible roles of plms: assessing difficulty, encoding semantic representations, directly producing sentences and evaluating their quality, all of which could be investigated further on the basis on their applicability in ai-supported language learning and other fields in education technology. from the feedback and data collected from the human evaluation, we can point out the potential for improving and combining these systems to balance their shortcomings, even though the retrieval methodology was considered to be the best in terms of diversity and adherence to difficulty level. the challenge of evaluating generated text prompted us to explore a state-of-the-art llm's ability in rating sentence quality. in our opinion, it is a promising direction because the model seems to be able to evaluate linguistic features of sentences. we found good agreement in rating text difficulty, but since each person could make different assessments, finding a way to take that variability into account could be useful for personalization. it could be studied whether using word-level features can prevent unknown kanji from appearing in example sentences. such features could be jlpt labels or the school grade level they are taught in. another research challenge is estimating the real vocabulary known by the learner, modeling the process of second language acquisition @xcite . additionally, there is potential for suggestion and generation of material based on each learner's interests. a direction to explore further is to experiment with more advanced llm prompting strategies, such as chain of thought or reinforcement learning, to iteratively refine outputs for better adaptation to learners' preferences. a retrieval approach like ours could serve as a starting point.