Starting from “Zero”: An Incremental Zero-shot Learning Approach for Assessing Peer Feedback Comments (2022)

Abstract: Peer assessment is an effective and efficient pedagogical strategy for delivering feedback to learners. Asking students to provide quality feedback, which contains suggestions and mentions problems, can promote metacognition by reviewers and better assist reviewees in revising their work. Thus, various supervised machine learning algorithms have been proposed to detect quality feedback. However, all these powerful algorithms have the same Achilles’ heel: the reliance on sufficient historical data. In other words, collecting adequate peer feedback for training a supervised algorithm can take several semesters before the model can be deployed to a new class. In this paper, we present a new paradigm, called incremental zero-shot learning (IZSL), to tackle the problem of lacking sufficient historical data. Our results show that the method can achieve acceptable “cold-start” performance without needing any domain data, and it outperforms BERT when trained on the same data collected incrementally.

## introduction peer assessment is a process whereby students assess other students' assignments by writing review comments against a set of assessment criteria provided by the instructor. this pedagogical strategy has been extensively applied across various academic fields and has demonstrated its effectiveness over the past decades @xcite . furthermore, peer assessment serves as a crucial tool for delivering necessary feedback in massive open online courses (moocs), as this assessment strategy allows moocs to scale up the feedback process while minimizing ongoing support costs. nevertheless, the benefits of peer assessment can only be achieved with quality peer feedback @xcite . course staff can manually review the credibility of each submitted feedback, but this is very inefficient. hence, there has been a surge of interest in automating the assessment of feedback quality by machine-learning algorithms. these algorithms typically assess quality by determining whether the feedback comprises certain features (e.g., contains "suggestion" and "problem" statements) @xcite . if those characteristics are not present in the submitted reviews, the peerassessment system could suggest that the reviewer revise the feedback to add the missing features. although these machine-learning algorithms for assessing feedback quality are very effective, they all have the same achilles' heel: dependence on enough domain-specific peer-feedback data. that is, for each new discipline, it takes several school terms to collect sufficient data before the model can be applied. thus, a desideratum of peer-assessment platforms is an effective quality-assessment model that does not require domain-specific historical data in "cold-start" condition (i.e., no domain data is available for training). additionally, this model should be capable of using incrementally collected data to progressively improve its performance. in this paper, we present an approach, named incremental zero-shot learning (izsl), for addressing lack of historical data in automated feedbackquality evaluation. the core idea of the method is to treat the problem of detecting quality feedback as a natural language inference (nli) task and utilize the pre-trained bart-based nli model @xcite to assess feedback quality. our results show that izsl can achieve acceptable performance in the "cold-start" condition on different datasets, and izsl can substantially outperform bert @xcite after training on the same incrementally collected data. the rest of the paper is organized as follows: section 2 presents related work. section 3 describes datasets. section 4 elaborates on our izsl method for assessing feedback quality. section 5 presents experimental results. section 6 concludes the paper and provides some discussion about future work. ## 1 the design is written great. it will be better to explain more about the pattern used. 2 related work ## dataset we captured data from expertiza. in this system, learners can submit their work and write feedback comments on peers' submissions based on a set of rubric prompts. for example, each reviewer might be asked to provide a comment for the criterion, "does the design incorporate all of the functionality required?" in this paper, the terms "feedback comments," "review comments," and "peer feedback" are used interchangeably to mean the textual responses to criterion in the rubric. we obtained two datasets from the aforementioned peer-review platform for this study. the first dataset, cs-peer-feedback, is derived from a graduate-level object-oriented development course. this dataset consists of 12,053 data points and is mildly imbalanced. the second dataset, ed-peer-feedback, comes from a graduate-level education course. the dataset contains 172 data points and is also mildly skewed. some sample peer-feedback comments are displayed in all feedback comments have been manually annotated by a fluent english speaker who is familiar with the course context. to measure the reliability of the labels, we randomly sampled 100 comments from each dataset and asked a second annotator to annotate them. we measured the inter-annotator agreement on each set of 100 randomly selected samples using cohen's κ coefficient. the average κ scores for the cs-peer-feedback dataset and the ed-peer-feedback data were 0.88 and 0.85, respectively. these scores suggest that the annotations are reliable (cohen's κ > 0.81 @xcite ). ## conclusion and future work the quality of peer feedback plays a vital role in peer assessment. however, lacking historical data for new curricula is a persistent problem. our work proposes a novel method for assessing feedback quality by converting it into an nli problem. the approach can potentially be generalized to other pedagogical tasks. future plans include investigating how to improve "cold-start" performance.