Your Answer is Incorrect... Would you like to know why? Introducing a Bilingual Short Answer Feedback Dataset (2022)

Abstract: Handing in a paper or exercise and merely receiving “bad” or “incorrect” as feedback is not very helpful when the goal is to improve. Unfortunately, this is currently the kind of feedback given by Automatic Short Answer Grading (ASAG) systems. One of the reasons for this is a lack of content-focused elaborated feedback datasets. To encourage research on explainable and understandable feedback systems, we present the Short Answer Feedback dataset (SAF). Similar to other ASAG datasets, SAF contains learner responses and reference answers to German and English questions. However, instead of only assigning a label or score to the learners’ answers, SAF also contains elaborated feedback explaining the given score. Thus, SAF enables supervised training of models that grade answers and explain where and why mistakes were made. This paper discusses the need for enhanced feedback models in real-world pedagogical scenarios, describes the dataset annotation process, gives a comprehensive analysis of SAF, and provides T5-based baselines for future comparison.

## introduction assessment and feedback are essential to highquality education @xcite . they allow learners and teachers to discover misconceptions, gaps in knowledge, and improvement opportunities. however, manually assessing learners' knowledge and providing helpful feedback is time-consuming and requires pedagogical as well as domain expertise. here, automatic assessment can free up teachers' time to focus on tutoring learners or adequately preparing classroom activities. moreover, it can be an alternative to peer-grading when course participant numbers increase beyond the financial feasibility of manual grading @xcite , making it particularly useful for freely accessible online courses. besides being cost-and time-efficient, automating assessment also offers unique teaching opportunities. as long as systems give individual, responsespecific feedback, learners may retry or take additional assignments and receive instantaneous feedback as often as they need. additionally, knowing that a system instead of one's teacher or professor will evaluate one's assignment can also reduce anxiety and help learners focus on their work instead of worrying about their reputation @xcite . therefore, it is unsurprising that automatic assessment has been an active research field over the past decades @xcite @xcite . so far, significant progress has been made. in particular, transformer models are approaching human experts' performance on specific datasets in the automatic short answer grading (asag) field @xcite . these models are trained to evaluate whether natural language responses fully answer open knowledge questions and typically output a score or label indicating the response's correctness. this kind of feedback is also called verification @xcite . an example can be seen in what are the challenges of mobile routing compared to routing in fixed and wired networks? please name and describe two challenges. ## related work while elaborated feedback datasets on language learning @xcite @xcite appeared recently, they focus on linguistic mistakes, such as grammatical errors, instead of content. our extensive literature review did not reveal datasets that included content-focused elaborated feedback on short answer responses. however, saf's feedback can be viewed as a textual explanation of the assigned score. therefore, comparable nlp datasets with textual explanations and publicly available asag datasets without explanations are discussed in the following sections. ## short answer feedback dataset (saf) to remedy the lack of content-focused elaborated feedback datasets, we provide saf, an english and german short answer dataset with explanations that serve as elaborated feedback. in total, the corpus contains 4,519 submissions similar to the example in ## experiments the goals of our experiments are threefold. firstly, we want to provide baselines for the dataset. for this reason, it makes sense to report a wide range of metrics future work may want to utilize. secondly, we hypothesize that including the question in the model's input would increase performance. typically, only the student and reference answers are compared in asag @xcite even though the question may contain additional important information. to investigate the question's effect on performance, we run each experiment in two settings: with a student and reference answer pair as model input or with a question, student, and reference answer triplet. finally, we want to explore the synergy between the asag scoring and classification tasks and feedback generation. we believe that grading and feedback should be trained jointly since the feedback should match the assigned grade @xcite , and both tasks benefit from extracting the same information from the answers. for example, a span of tokens negatively impacting the grade should also affect the feedback accordingly. our experiments investigate the hypothesis that feedback generation benefits more from being paired with the more informative asag scoring task (0-1) than the verification feedback label classification (correct vs. incorrect vs. partially correct). ## conclusion and future work this paper introduces the elaborated feedback generation task. we provide a benchmarking dataset containing short answers, scores, and textual explanations of given scores to kick off this task. as of yet, the dataset consists of 4,519 submissions to german and english questions. we demonstrate saf's reliability with high inter-annotator agreements. in section 3.3, we presented aspects of the dataset we plan to improve. while the dataset is sizable for a manually annotated task of this complexity, it is small compared to other nlp tasks' crawled, large-scale datasets. we plan to mitigate this by incorporating additional questions in future iterations of the dataset. the focus will be on more complex questions to improve the class balance and questions of other domains and languages to increase diversity. the model's ability to general-ize to unseen questions may also benefit from a more diverse dataset. we also observed that common text similarity metrics can provide a valuable first impression of the feedback's quality but are not sufficient to fully capture it. thus, we would recommend including humans in the evaluation loop. a possible evaluation setup could ask annotators whether the generated feedback expresses the same meaning as the reference feedback included in the dataset. we believe annotators could also carry out this task with limited background in the provided domains. nevertheless, we provide the detailed scoring rubrics utilized by our annotators along with the dataset to support future human evaluations. finally, the baselines presented in this paper can be improved. considering the deep understanding human graders require for this task, we believe neuro-symbolic approaches to be an exciting avenue of future research. current models may especially benefit from incorporating knowledge bases and other reference material.