J apanese- E nglish Sentence Translation Exercises Dataset for Automatic Grading (2024)

Abstract: This paper proposes the task of automatic assessment of Sentence Translation Exercises (STEs), that have been used in the early stage of L2 language learning.We formalize the task as grading student responses for each rubric criterion pre-specified by the educators.We then create a dataset for STE between Japanese and English including 21 questions, along with a total of 3,498 student responses (167 on average).The answer responses were collected from students and crowd workers.Using this dataset, we demonstrate the performance of baselines including a finetuned BERT model and GPT-3.5 with few-shot learning. Experimental results showed that the baseline model with fine-tuned BERT was able to classify correct responses with approximately 90% in F_1 , but only less than 80% for incorrect responses. Furthermore, GPT-3.5 with few-shot learning shows a poorer result than the BERT model, indicating that our newly proposed task presents a challenging issue, even for the state-of-the-art large language model.

## introduction sentence translation exercises (stes) are often utilized as educational tools in the early stages of l2 language learning, particularly between language pairs that are linguistically distant from each other @xcite . figure 1 shows an example of ste. here, a learner translates a short sentence in their native language (l1) into the language they are learning (l2), and these translations are graded following analytic criteria within the grading rubric such as e3 and g4, which correspond to specific grammar items or expressions. ## sentence translation exercise (stes) dataset to implement the automatic ste scoring, we introduce an ste dataset. this dataset currently com-prises 21 japanese-to-english ste questions with detailed rubrics and annotated student responses. these questions and the scoring rubrics were created by specialists in the design of english learning materials. the questions were constructed to cover all the major grammar topics in several well-known english textbooks used in japanese high schools. essentially, stes are designed such that they limit variations in correct responses from the outset. in practical settings, however, educators may adjust the grading rubric by incorporating variations in correct responses, previously unidentified during the rubric's initial creation, to accurately evaluate the student responses. to replicate this process, we initially create the analytic criteria, followed by the collection of student responses as described in the following subsection. subsequently, we refine the rubric by reviewing the collected responses, to preempt any challenges that may arise during the grading procedure. in the following sections, we will discuss in detail the methods used to gather responses, as well as the annotation process, and statistically analyze the whole dataset. ## method we employ a bert @xcite )-based classification model and the gpt models (openai, 2023) with in-context learning as a baseline for our task formulation. this section discusses these baseline models in detail. ## experiments in the experiment, we investigate the feasibility of our task formulation for stes using the bert model and the state-of-the-art large language models, gpt-4 and gpt-3.5. we also investigate the impact of the number of in-context examples on the scoring performance. ## related work grammar error correction (gec) and short answer scoring (sas) are the two major research areas in the automatic evaluation of descriptive english responses. we position this study between these two research domains. ## conclusion this study introduced a novel task focusing on the automatic grading of sentence translation exercises (stes) for educational purposes. we formalized stes as a task of grading each analytic criterion predetermined by teachers' intentions and constructed a dataset to implement the task. this first-of-its-kind dataset emulates and reflects the practical form of l2 learning in the responses of learners. we also used finetuned bert and gpts with few-shot in-context learning to establish a baseline and demonstrate the feasibility of the formulated framework. in our experiment, although the gpt models showed substantial performance in various nlp tasks, they remained inferior to the bert model, suggesting that our newly defined task continues to be challenging even for the state-of-the-art llms, therefore necessitating further exploration. with regard to future direction, we are contemplating the integration of technologies such as gec and machine translation within our model. we aim to build cross-questions strategies to automatically identify expressions that diverge from a provided rubric while preserving the text's fundamental meaning using a combination of these technologies. for this purpose, our plan involves further subdividing the ste grading task and leveraging llms to address each minimized task such as correcting grammatical errors, assessing the consistency of meaning with l1, and identifying expressions aligned with the learning objectives in each exercise. this approach also aims to investigate tasks where llms may not excel in ste scoring and enhance their overall performance. additionally, in an educational context, we also consider generating more comprehensive feedback comments on the scoring results, extending beyond the estimation of justification cues.