The Specificity and Helpfulness of Peer-to-Peer Feedback in Higher Education (2022)

Abstract: With the growth of online learning through MOOCs and other educational applications, it has become increasingly difficult for course providers to offer personalized feedback to students. Therefore asking students to provide feedback to each other has become one way to support learning. This peer-to-peer feedback has become increasingly important whether in MOOCs to provide feedback to thousands of students or in large-scale classes at universities. One of the challenges when allowing peer-to-peer feedback is that the feedback should be perceived as helpful, and an import factor determining helpfulness is how specific the feedback is. However, in classes including thousands of students, instructors do not have the resources to check the specificity of every piece of feedback between students. Therefore, we present an automatic classification model to measure sentence specificity in written feedback. The model was trained and tested on student feedback texts written in German where sentences have been labelled as general or specific. We find that we can automatically classify the sentences with an accuracy of 76.7% using a conventional feature-based approach, whereas transfer learning with BERT for German gives a classification accuracy of 81.1%. However, the feature-based approach comes with lower computational costs and preserves human interpretability of the coefficients. In addition we show that specificity of sentences in feedback texts has a weak positive correlation with perceptions of helpfulness. This indicates that specificity is one of the ingredients of good feedback, and invites further investigation.

## introduction with thousands of students in moocs and hundreds of students in university classes, instructors increasingly apply the approach of peer-to-peer feedback (or, 'peer feedback'), where students provide feedback to their peers @xcite . peer-feedback enables instructors to provide individual feedback on every piece of coursework by leveraging the potential of students to provide feedback to each other @xcite . nevertheless, since students are often not experts in providing feedback, the instructors need to ensure that the feedback is helpful @xcite . research has shown that one factor determining feedback helpfulness is whether the feedback points are generic or specific @xcite @xcite . generic feedback such as "improve your submission" are less helpful than detailed, targeted advice such as "add a timeline" or "change the caption in figure 1 ". however, the challenge is that instructors who do not have time for providing feedback themselves also do not have time for checking the specificity level of peer feedback @xcite . one approach is to develop a model which automatically analyses feedback specificity using natural language processing. recent work has been carried out into automatic classification of sentence specificity in newspaper articles; for instance by @xcite , @xcite and @xcite . our work builds on this previous research and at the same time provides distinct contributions. firstly, we apply the approach in the novel domain of education and peer-feedback, which is inherently different in its purpose and nature compared to the news domain which features in previous work. news articles are written for a general audience with the purpose to inform, whereas peer-feedback texts are written to reveal the strengths and weaknesses of written work and provide suggested improvements. furthermore, in the peer-feedback scenario, each student has put effort into their assignments: thus they have a certain expectation as to the quality of feedback they ought to receive. secondly, we have developed a unique dataset of peer-feedback containing more than 1000 sentences labelled for specificity. thirdly, the data we work with are in the german language: to the best of our knowledge, all previous related work has been on english. fourthly, we find that there is a correlation, albeit weak, between sentence specificity and the perceived helpfulness of peerfeedback. we train and evaluate four classifiers based on a feature set which is determined by methods described in previous work and our own observations of specificity in peer-feedback texts. we also explore the relationship between sentence specificity and perceived helpfulness of peer-feedback, finding a weak positive correlation, which suggests that specific sentences are helpful but also that further work is needed to uncover the other ingredients of good feedback. we contribute our collected corpus of sentences from peer-feedback texts in german for further analysis and hope to provide researchers and practitioners with a detailed analysis and discussion of sentence specificity. the code and annotated corpus can be accessed via github foot_0 . ## data our dataset consists of peer-feedback texts written by students on a masters course on business innovation at a german-speaking university, collected over the past five years. students followed a peerfeedback process which is similar to the scientific paper review process in academia @xcite . students submitted their assignment to a learning management system. each assignment was afterwards anonymously distributed to three reviewers who each wrote their feedback before then being sent back to the assignment author. there were no rules on how to write the feedback, students only received three guiding questions: what was good, what was not so good and what possible improvements could be made? each feedback text is on average 250 words long. specific feedback gives the recipient a more direct indication of strengths, weaknesses, and suggested changes (e.g. [s1] and [s2]). general sentences such as [g1] and [g2], on the other hand, often refer to entire sections or the whole work and require further clarification-questions or interpretation by the feedback recipient. note that peer-feedback has unique characteristics which differ from other domains. it is possible for sentences to contain generalized statements which would normally be classified as such, yet in the context of peer review feedback they are in fact specific suggestions. for instance the sentence, "young people are much less obsessed with their car's internal specs than older people", contains a rather generalized statement. yet in the context of a reviewer critiquing the reviewee's business personas, it may appear to be more specific: "i do not think the persona of anna would be interested in your service, because young people are much less obsessed with their car's internal specs than older people.". this more complex sentence becomes a more specific criticism than simply stating, "i don't think that the persona of anna is realistic". for the annotation process we randomly sampled 1000 feedback texts from our corpus and adopted two strategies for annotation. first, relying on many annotators who rated only a limited amount of sentences, whereby each sentence is annotated by 5 annotators and second, relying on two students who in several workshops receive training on how to annotate specificity and an expert in nlp as arbitrator for the two annotations. in both strategies the annotators rated the specificity on a scale of 1 (very general) to 5 (very specific) developed by @xcite and @xcite . we chose the two strategies because, both have their advantages and disadvantages. for example, the first approach reduces systemic bias of one individual annotator on the whole dataset, since annotators only labelled a limited number of sentences. a downside of this strategy is that, there is no opportunity for annotators to learn over time and therefore reaching agreement on the level of specificity for one sentence is more difficult. the second strategy has the benefit of learning effects but the possible downside of systemic biases by two annotators labelling many sentences. for the first strategy, we used survey circle foot_1 . the dataset was formed from a random sample of 1000 sentences from the 1000 feedback texts. we made the annotation job available to survey circle users based in germany, austria, switzerland, specifying that they should be german speakers. the users on survey circle are typically students from a variety of disciplines. overall, 1000 sentences where annotated by 200 users who each annotated 25 sentences. each sentence was reviewed five times by five different annotators. for quality control, we removed ratings by users who chose the same label for every one of their sentences, and who did not complete at least 15 annotations. since our focus was on high quality data we only chose sentences with an inter-annotator agreement (iaa) higher than 60% to further proceed with in our classification algorithm, leaving us with 331 sentences with an average iaa of 0.804. to create a final dataset, we took the mean of 5 annotations, which resulted in the final specificity score. the fact that we had to filter out so many sentences at this stage, due to low iaa, prompted us to try a different approach to annotation. in the second strategy, we randomly selected 75 of the 1000 feedback texts and removed all sentences having a character length lower than 40 (since usually those sentences solely included bullet points, enumerations, or wrong sentence segmentations). this pre-processing resulted in a final dataset of 800 sentences. two native german speakers annotated the sentences independently from each other in the same manner as done previously on survey circle, but this time using the decision tree shown in figure 1 . a team workshop and several calibration training sessions were performed to reach a common understanding of the annotation. 800 feedback texts were annotated by figure 1 : specificity annotator prompt. q1: "is this feedback sentence only applicable to this individual work (eg. it references specific paragraphs, objects or people from the source text) or could it be written generically about many different works?" q2: "can this feedback sentence stand on its own, or does it require concretising questions or interpretation by the feedback recipient, in order to be implemented or understood?" the two annotators -in case of disagreement an expert arbitrator was consulted in order to discuss the specific cases in detail and to reach an agreement between the two. the annotations resulted in an iaa of 0.746. to create a single version of the gold standard, the arbitrator took the final decision in cases where the two annotators still disagreed. finally, we merged both datasets to give a total of 1131 sentences. for our machine learning experiments we additionally obtained a binary label for each sentence, therefore we aggregated the ratings 4 and 5 to be specific (meaning a label of 1) whereas ratings 1 and 2 were deemed to be general (i.e. a label of 0). we chose to remove from the dataset the 170 sentences that received a rating of 3 "unrateable". this gave us an almost balanced dataset containing 48% general sentences and 52% specific sentences. ## classification experiments using the data described in the previous section we undertake experiments to automatically classify sentences as specific or general. being able to do so accurately will allow us to identify when a reviewer's text contains no specific feedback, and potentially encourage them to be more specific in downstream applications. we compare 'classic' feature-based classifiers with a bert-based model @xcite fine-tuned on our dataset. ## results in to summarise, we find that the bert-based finetuned classifier performs best. not unexpectedly, the superior performance of bert comes at a computational cost, as the fine-tuning of the transformer takes significantly longer than fitting the other models (>5mins as opposed to a few seconds), and requires gpu. furthermore, bert offers little in the way of interpretability. in this regard, algorithms such as logistic regression and random forests are advantageous due to their human understandable coefficients. we would therefore opt for a featurebased classifier if putting a sentence specificity detection system into production: the efficiency gains and advantage with respect to explainability in our view outweigh the performance boost provided by a bert-based model. we analysed which of our features were the best predictors of sentence specificity. to that end we rank the features from the logistic regression classifier by coefficient. the coefficients represent log odds that an observation is in the target class ('specific'), and thus we take the exponent of the coefficients to obtain odds ratios. we find that features relating to numbers and currency, non-alphanumeric characters, and named entities are the most likely to occur in specific feedback. this reflects the fact that the subject domain is business but also that such features are associated with specific references to locations in the text, and the non-alphanumeric characters featuring in specific feedback formatting such as bullet points, section markers and parentheses, or punctuation used as connectives (e.g. right arrows and dashes). we find that other highly weighted features are representative of specific feedback texts in general, such as a high number of noun chunks, named entities, words with clear polarity, adjectives and discourse connectives. finally, we note that a longer minimum hypernym path in germanet for words in a sentence is associated with more specific feedback, as we hypothesised (section 4.1). ## feedback specificity and helpfulness we examined the interplay between feedback specificity and helpfulness to evaluate the hypothesis that more specific feedback is more helpful @xcite . we sampled 500 feedback texts from the business masters course previously referred to, presented them to survey circle annotators (students and phds), and asked them to score the strength of their agreement with the following four statements on a scale of 1 to 10 for each text: "the feedback from the reviewer was helpful","the reviewer was able to provide constructive suggestions on their stated critical aspects", "the reviewer was able to identify critical aspects in the assignment", or "the feedback from the reviewer was of high helpfulness". the mean of these likert scores was taken from 5 annotators per text and across all 4 statements to give an overall feedback helpfulness score for each text between 1 and 10. to derive a specificity score for a feedback text, we made per-sentence specificity predictions using the bert-based model trained on the annotated peer-feedback set of 1000 sentences described above. the score per text was then the average sentence specificity prediction, a value between 0 and 1. the correlation between text specificity scores and helpfulness ratings showed a correlation of 0.21 with a statistically significant p-value <.001. this finding helps to corroborate the hypothesized relationship between specificity and feedback helpfulness, while reminding us that the relationship is not straightforwardly linear. a strongly helpful feedback text should not contain entirely general sentences or entirely specific ones, but some combination of the two. in figure 2 we show a scatter plot of the feedback specificity per text (per cent of sentences in a text classified as specific by the model) against the feedback helpfulness score per text calculated in the way described above, and both the weak correlation and variation in the relationship are apparent. ## discussion we show that sentence specificity can be classified successfully in german peer-feedback texts. this can be a useful first step for various education technology applications. for instance we can provide students with automated advice on how to improve their written peer-feedback. it can potentially help with feedback to students on their written assignments as well, in cases where students have not made sufficiently specific statements. for this reason, explainability and low computational cost are important factors in weighing up the performance of our feature-based and bert-based specificity classification models. one limitation of the current sentence level approach is that it fails to deal with dependent sentences where a feedback point is argued for over multiple sentences. to accurately rate the specificity in such cases, it can be crucial to take into account the context in which a sentence appears. consider, for instance, the following example: to address this issue in future work, we can attempt to segment texts into 'argumentation chunks' rather than sentences. such an approach requires a combination of information density extraction, argumentation mining and specificity prediction. this observation is congruent with previous work which concluded that context information should be considered in the annotation procedure to mitigate the effect of anaphoric and topical references that may otherwise be inadequately dealt with @xcite . in addition, it is apparent that any downstream application should be tuned so that recommendations on feedback specificity at a per-sentence level take the whole text into account, so that the student is encouraged to write a well structured mix of general and specific feedback. finally, we note that specificity could be just one of multiple components that determine the helpfulness of feedback. in truth, feedback helpfulness is difficult to measure objectively since in large part it is driven by how helpful a student perceives it to be. @xcite state that, "what a student considers good assessment and feedback is shaped by the assumptions they hold as to the nature and certainty of knowledge (baxter magolda, 1992), their prior learning experiences (o'donovan, 2017) as well as the timing of their consideration @xcite )". just getting technical factors right will not ensure student satisfaction with feedback (p. 8)". in the long run, the sole focus on the feedback itself and its language is too narrow as it is only part of the complexity of providing good feedback @xcite . to make a holistic improvement to feedback procedures at large as well as enhance student engagement and satisfaction, peer assessment process design, pre-feedback con-ditions, and predictability need to be considered as well @xcite . it is likely that perceptions of feedback helpfulness are influenced by a number of contributing factors, some of which are in the text -e.g. lexical content, pragmatic implication and argumentation -while others are external and concern the wider educational context of the assignment. for instance, the feedback should be relevant to the task, on topic, and consistent with the curriculum. we expect that specific sentences should also be used with more generic 'big picture' and bridging sentences, and that feedback providers could be prompted to provide a mixture of both. there is also the pedagogical question of timing: when more specific feedback is beneficial for the student and when it is not. these issues represent opportunities for future investigation. ## conclusion we have presented experiments in automatic classification of the specificity of german sentences in peer feedback written by students in an online assignment reviewing system. we derived features based on previous work and the qualitative analysis of our dataset, and performed multiple experiments using machine learning models compared to a transfer learning approach with bert @xcite . we found that our classifiers were able to successfully predict sentence specificity with an accuracy of at least 70% for all models. the bert model mostly outperforms the feature-based classifiers, but it has the highest computational cost and does not have human interpretable coefficients. svm performs best on the peer-feedback texts for feature-based models, is computationally more efficient and provides per-feature coefficients which enable downstream explainability for any user-facing system. in addition, in the analysis of our logistic regression model we report which features are most likely to indicate feedback specificity, and find that numbers, noun chunks and non-alphanumeric characters are at the top of the list. we found a weak correlation between crowdsourced assessments of feedback helpfulness and feedback specificity, underlining that texts containing relatively high proportions of specific sentences are more likely to represent good quality feedback.