Reviewriter: AI -Generated Instructions For Peer Review Writing (2023)

Abstract: Large Language Models (LLMs) offer novel opportunities for educational applications that have the potential to transform traditional learning for students. Despite AI-enhanced applications having the potential to provide personalized learning experiences, more studies are needed on the design of generative AI systems and evidence for using them in real educational settings. In this paper, we design, implement and evaluate \texttt{Reviewriter}, a novel tool to provide students with AI-generated instructions for writing peer reviews in German. Our study identifies three key aspects: a) we provide insights into student needs when writing peer reviews with generative models which we then use to develop a novel system to provide adaptive instructions b) we fine-tune three German language models on a selected corpus of 11,925 student-written peer review texts in German and choose German-GPT2 based on quantitative measures and human evaluation, and c) we evaluate our tool with fourteen students, revealing positive technology acceptance based on quantitative measures. Additionally, the qualitative feedback presents the benefits and limitations of generative AI in peer review writing.

## introduction peer reviewing is a process by which learners provide formative feedback to each other on an individual task based on assessment criteria @xcite . research has found theoretical and empirical evidence for the positive effects of peer reviews on critical thinking skills @xcite , communication skills @xcite , and learning motivations @xcite . the prevailing practice of peer review in tertiary education is evident in the eruption of massive open online courses (moocs) @xcite . in these large-scale learning scenarios, peer review is particularly important since it is challenging for teachers to give effective oneby-one feedback due to immersive workload and shortage of time @xcite . however, according to @xcite , a challenge that plagues many student writers, including those having satisfactory grammar and spelling skills, is writer's block. it was defined by @xcite as "that frustrating, self-defeating inability to generate the next line, the right phrase, the sentence that will release the flow of words again." a collaborator who provides instructions and points out new directions might help alleviate writer's block @xcite and the combination of a writer's own ideas with suggested ideas is a form of psychological creativity @xcite . novel llms have the potential to address the challenge of writer's block by generating suggestions for the next lines, right phrases, or sentences, thereby facilitating the flow of ideas @xcite , and helping students compose responses more efficiently @xcite . there are llm-based collaborative writing tools to provide support for various writing tasks, including story writing @xcite , science writing @xcite , and screenwriting @xcite . however, few have investigated the utilization of generative ai for peer review writing tasks. therefore, in this paper, we build and evaluate reviewriter which can provide ai-generated instructions tailored to students' needs while writing peer reviews. it suggests possible directions based on students' input to inspire divergent outcomes while still leaving learners in control of the final text. to investigate how to provide students with help to overcome writer's block in peer review writing, we conduct a literature review to gather insights for a peer review support system. we summarize five user requirements from interviews with twelve graduate students. based on those, we develop seven design principles for providing aigenerated instructions in peer review tasks. next, we search peer review corpora satisfying certain criteria and pre-process 11,925 student-written peer derive insights and requirements from literature and design interviews (@xmath0 evaluate reviewriter with student from higher education using a mixed method study (@xmath1 ## discussion peer review writing is an increasingly important educational task in large-scale or distance learning scenarios since it enables personalized feedback to be delivered at scale, thereby lessening the workload of instructors @xcite and boosting learners' motivation @xcite . however, during writing peer reviews, students may experience obstacles such as writer's block @xcite where they struggle to generate the next line, the right phrase, or the sentence oliver @xcite . llms can help to overcome this obstacle by producing adaptive instructions based on students' input, which ultimately aid in the seamless progression of thoughts @xcite . to do so, we develop a novel peer review writing tool called reviewriter. it allows students to use aigenerated instructions as an inspiration and incorporate those ideas into their own work in a creative and original way, such as by adapting, mixing, or reinterpreting those instructions @xcite . our study contributes at least three key aspects to the innovative use of nlp in education. first, we explore the personalization of ai-generated instructions in a specific pedagogical scenario -peer review writing @xcite by gathering insights from literature review and student interviews @xcite . second, in contrast to @xcite which used gpt-3 without adaptation for collaborative writing, we fine-tune three german language models on a corpus selected based on certain criteria to provide specialized content with high quality. afterward, we choose german-gpt2 based on quantitative measures and human evaluation. third, as noted in @xcite , "lack of feedback" is one of the challenges to using generative models in education. therefore, we evaluate our tool with fourteen students and the result reveals positive technology acceptance based on quantitative measures. through our qualitative evaluation, we find that students generally enjoyed seeing generated instructions with varied content to spark ideas. and they were enthusiastic and excited about writing with generative language models. we recognize that there is a need for further research on the effectiveness of llm-based writing support tools in various contexts, as well as the improvement of faithfulness and factuality in ai-generated instructions @xcite . nonetheless, our study contributes to the growing body of knowledge on the potential of generative ai to provide personalized writing instructions and enhance students' learning experiences @xcite . ## conclusion and future work to help students mitigate writer's block during peer review writing, we design, build, and evaluate reviewriter, a novel tool that aims to provide students with ai-generated instructions during their peer review writing process. we provide design insights with pedagogical considerations of integrating llms into peer-review writing systems. our evaluation involves fourteen students from tertiary education, who reported enjoying the interaction with the system, finding it easy to use, and expressing interest in using similar tools in the future. they also pointed out that the relevance of the generated instructions could be further improved. we present reviewriter, including its design rationales and evaluated interface, as a contribution to the exploration of llms' potential in innovative nlp-based approaches in education. as nlp continues to advance, we aspire that our work will encourage other researchers to explore how generative ai can be integrated into educational applications to benefit teachers and students, while promoting responsible and ethical use. for future work, we will investigate students' perceptions of peer reviews from different sources: their peers, peers using reviewriter, and entirely ai-generated reviews. we will collect ratings and feedback from students who receive these reviews and compare the relevance, quality, and usefulness of the texts generated from each source. additionally, we aim to integrate reviewriter into the university's existing peer review system, enabling widespread adoption among students across various courses. by incorporating ai-generated instructions into routine peer reviews, we can examine the long-term impact on student's writing skills, critical thinking abilities, and overall academic performance. to enhance the relevance of the ai-generated instructions in reviewriter, we will refine the algorithms and models based on feedback from our evaluation participants. our iterative development process will involve incorporating more contextual information, employing advanced nlp techniques, and leveraging user feedback to achieve higher accuracy and helpfulness in the ai-generated instructions.