Automated Focused Feedback Generation for Scientific Writing Assistance (2024)

Abstract: Scientific writing is a challenging task, particularly for novice researchers who often rely on feedback from experienced peers. Recent work has primarily focused on improving surface form and style rather than manuscript content. In this paper, we propose a novel task: automated focused feedback generation for scientific writing assistance. We present SWIF ^2 T: a Scientific WrIting Focused Feedback Tool. It is designed to generate specific, actionable and coherent comments, which identify weaknesses in a scientific paper and/or propose revisions to it. Our approach consists of four components - planner, investigator, reviewer and controller - leveraging multiple Large Language Models (LLMs) to implement them. We compile a dataset of 300 peer reviews citing weaknesses in scientific papers and conduct human evaluation. The results demonstrate the superiority in specificity, reading comprehension, and overall helpfulness of SWIF ^2 T’s feedback compared to other approaches. In our analysis, we also identified cases where automatically generated reviews were judged better than human ones, suggesting opportunities for integration of AI-generated feedback in scientific writing.

## introduction peer feedback is essential in the scientific communication process, facilitating the development, interpretation, and dissemination of new findings @xcite . yet, delivering high-quality feedback is both time-intensive and demanding @xcite . in response to this challenge, scientists are increasingly utilizing tools such as grammarly foot_1 , to improve the clarity of their manuscripts @xcite . however, these tools focus primarily on enhancing expression clarity rather than refining the research content, and it has been argued that a scientific peer should prioritize scientific value over editorial comments to enhance the quality of a submission @xcite . given the importance of peer feedback in the scientific process, predicting peer response would be a powerful assistive tool for writers. several studies @xcite have explored automated scientific reviewing. while the task is similar to scientific writing assistance, these studies focus on extracting high-level information for paper-level reviews, lacking specificity. additionally, their emphasis is often on predicting conference acceptance rather than revising the manuscript. more recently, @xcite investigated the usefulness of gpt-4-generated comments on entire scientific papers. their results indicate that, while such feedback benefits researchers, a crucial limitation lies in the models' inability to generate specific and actionable suggestions. in this paper, we introduce the task of automated focused feedback generation for scientific writing assistance (figure 1 ). focused feedback entails providing specific, actionable, and coherent comments that identify weaknesses in a paper or suggest revisions. to this end, we present swif 2 t, a scientific writing focused feedback tool. this approach leverages multiple llms to decompose the process of generating scientific feedback into four components (figure 2 ): planner, investigator, reviewer and controller. the planner designs a step-by-step plan involving the other components. it first instructs the investigator to answer questions using the paper or literature to enrich the paragraph under review with relevant context. then, it directs the reviewer to leverage the gathered context to identify a specific portion requiring review, predict a weakness type (e.g. originality in figure 1 ) before generating focused feedback accordingly. the controller manages the plan's progress and coordinates other models' actions. recognizing the pivotal role of the plan in shaping the review outcome, we also propose a plan re-ranking method aimed at optimizing the coherence, structure, and specificity of the generated plan, thereby enhancing the effectiveness of our scientific feedback generation process. to assess our system, we compile a test set using human-written feedback derived from peer reviews. it consists of 300 examples where reviews highlight weaknesses in specific paragraphs of scientific papers. we conduct both automatic and human evaluations of swif 2 t, comparing it against robust baselines such as cove @xcite and gpt-4. the results demonstrate that swif 2 t outperforms other models not only in terms of similarity to human-written reviews, but also in specificity, reading comprehension, and overall helpfulness of its comments. our analysis reveals instances where swif 2 t -generated comments are preferred over human-written ones, hinting at the potential for integration of ai-generated feedback in the scientific writing process. finally, our analysis highlights the strengths and weaknesses of tested llm-based approaches, outlining avenues for future research. these datasets have facilitated the creation of tools that automatically generate reviews from papers @xcite @xcite , although these reviews tend to focus on the high-level information of a scientific paper given that the tools consume entire papers at once. recently, @xcite and @xcite investigated the potential of gpt-4 in assisting the peer review process. they found that gpt-4's feedback was comparable to human-written feedback in terms of helpfulness. however, they emphasized its limitations, notably its inability to provide specific, actionable feedback and to review a paper's low-level details. to address the lack of specificity in generated comments, d'arcy et al. (2024) introduced marg, a system that uses multiple llm instances to handle different sections of scientific papers. their proposed approach reduces generic feedback and increases comment quality. however, their focus is on generating paper-level reviews that emphasize the overall work rather than detailed aspects -which our study aims to address. text revision several works have developed text revision tools to rewrite the intended text in scientific style. a recent survey by @xcite has grouped text revision tools into three different categories. the first one is grammatical error correction (gec), which seeks to detect and rectify grammatically incorrect or misspelled text @xcite . the second category focuses on more substantial revisions than gec, such as changing the structure of a sentence and rephrasing for clarity to generate error-free, proofread versions of earlydraft sentences from scientific papers @xcite @xcite . the third category of text revision tools focuses on enhancing writing abilities through the visualization of the rhetorical structures in text @xcite . while these approaches provide highquality writing assistance, they do not target providing constructive criticism on the research content. more relevant to our research is d'arcy et al. (2023), which explores the tasks of aligning highlevel feedback to specific edits and generating revisions for scientific papers. this work differs from ours in that the presented tasks use reviews as input, whereas we focus on generating the reviews themselves. they discuss as a long-term goal the development of an "ideal writing assistant" that consumes relevant information from the paper and literature when generating an edit, which we implement in our work for scientific feedback generation. ## focused feedback generation we define focused feedback as feedback that comprehends the analysed paper (reading comprehension), clearly indicates what specific elements in the paper need revision (specificity), and communicates to the author which actions they can take to address the issues (actionability). these ultimately contribute to the overall helpfulness of the feedback, i.e. the accuracy of the points discussed in the review and the extent to which they could assist the author(s) in revising their paper. the envisioned model we aim to create concentrates on specific paragraphs p within a paper p to ensure no detail is missed. given that a paragraph on its own may lack context, the model requires access to the entire paper and relevant literature. this access enables the model to provide more accurate and contextualized feedback. the desired output is comments highlighting weaknesses in the paragraph and/or suggesting revisions to the paper. prior studies @xcite @xcite utilize aspects inspired by conference reviewing guidelines 3 to classify review sentences according to the aspects they address. in our study, we employ these aspects as weakness types to capture the focal points 3 example with definitions: https://acl2018.org/ downloads/acl_2018_review_form.html of reviews and exclude review sentences that do not constitute focused feedback during dataset compilation (section 5.1). specifically, we focus on the following five aspects 3 : replicability, originality, empirical and theoretical soundness, meaningful comparison, and substance. we exclude remaining aspects because they focus on refining the surface form (readability and clarity), are inherently subjective (impact, motivation) or do not constitute feedback (summary, appropriateness). ## swif 2 t 4.1 description our system (figure 2 ) relies on four components: planner, investigator, reviewer and controller. given a paragraph in a paper, the planner initially formulates relevant questions and instructs the investigator to address them using information from the rest of the paper or literature to retrieve relevant context (figure 2 ). for questions related to the literature, we use a search engine and feed retrieved pages into the investigator's qa pipeline. to answer a question utilizing the content of the paper or a retrieved document, this pipeline first splits the document into segments of equal lengths before creating embeddings for each. a similarity search is then performed to retrieve the top 5 text chunks that are most similar to the question. finally, our abstractive question-answering llm generates an answer based on the retrieved paragraphs, responding with "i don't know." when questions cannot be answered from the provided documents. then, once enough information has been gathered, the reviewer component utilizes it to identify a section of text requiring revision, predict an aspect to concentrate the feedback on before generating the focused feedback. after each step in the plan (example in figure 3 ), the controller decides whether the next step should be executed or replaced, enabling it to adapt the plan if needed. ## implementation 5.1 dataset compilation to construct our dataset, we utilize several existing peer review datasets that contain complete reviews and paper content: nlpeer @xcite , f1000rd @xcite , peerread @xcite , moprd @xcite , the dataset by @xcite , asap-review @xcite and aries (d' @xcite . we extract linked paragraph-review pairs from aries and a subset of f1000rd. for the remaining datasets without annotated links, we automatically retrieve all paragraph-review pairs where the review quotes a sentence within the corresponding paragraph. we implement a two-step data filtering process to ensure our reviews (1) identify weaknesses and/or propose revisions in a paragraph while (2) providing feedback on research and technical aspects. the first step involved actionability detection: we trained a model on the subset of f1000research containing reviews annotated with communicative purposes (recap, strength, todo, weakness, structure, other), applied this model to our collected reviews, and only retained those indicating weaknesses (weakness) or suggesting recommendations (todo). next, aspect annotation was performed on these actionable reviews. here, we trained a model on asap-review to predict the aspect addressed in our reviews from the following list: replicability, originality, empirical and theoretical soundness, meaningful comparison, and substance, summary, clarity and motivation before then only retaining those annotated with one of the five aspects highlighted in bold as motivated in section 3. by fine-tuning roberta models, we achieve high accuracy in pragmatic tagging (84.40% on test set, six labels) and aspect annotation (89.20% on test set, eight labels), enabling effective data filtering (implementation details in appendix b). the resulting dataset comprises 2,581 paragraphs paired with human-written reviews. for our test set, we first randomly selected 300 examples from our dataset. manual quality assurance was then conducted to ensure that our reviews are actionable, relevant and address one of the five aspects we focus on (section 3). we replaced inadequate reviews with randomly sampled instances from the dataset that met the criteria. for instance, reviews such as required revision in the abstract 'to our knowledge...standard manner' (lacking specificity about the revision needed) or the statement 'types of ...' is not clear. (misclassified as soundness instead of clarity) were excluded. however, we abstained from evaluating the helpfulness of individual reviews, recognizing the subjectivity of assessing this criterion, which could introduce bias into the selected dataset. in the final test set, 51.3% of examples were sourced from aries, 20.3% from asap-review and the remainder from other discussed datasets (dataset composition details in appendix a). ## evaluation 6.1 evaluation setup human evaluation is conducted on a subset of 100 machine learning and nlp examples from the test set. each model we consider is fed a paragraph and directed to identify a portion of text containing a weakness, predict a weakness type then generate a focused review. to guide our approach to human evaluation, we conducted several pilot studies. in shaping our evaluation criteria, insights from these pilot studies and previous research guided our selection. @xcite highlighted the lack of specificity in llm-generated reviews as a major weakness, emphasizing generic comments like "add more experiments on more datasets." to address this, we compare reviews based on their specificity and actionability, offering an indication of how often a system provides precise suggestions that allow authors to understand the necessary actions. secondly, pilot studies revealed a tendency for models to hallucinate in instances where context is lacking, e.g., suggesting the addition of information present in another paragraph of the paper or making unfounded assumptions. the reading comprehension criterion captures such instances. the third criterion assesses overall helpfulness, considering the accuracy of the point and potential impact on helping the author revise the paper. additionally, the pilot studies revealed that the task's inherent complexity posed challenges for annotators in providing absolute review rankings. as a result, we opted for pairwise comparisons to mitigate subjectivity. no information about the approach generating a particular review was disclosed, and the order of the reviews was randomized. as there were four systems and three criteria, one example involved 18 blind pairwise comparisons -one for each criterion and each pair. in reporting our results, we opted to utilize dominance scores, which represent the sum of the instances where one approach was ranked superior to another, enhancing transparency regarding the frequency of such rankings based on specified criteria. furthermore, we considered the task's difficulty and estimated the time required to annotate an example. based on the pilot studies, we found that completing one example, i.e., understanding the paragraph and reviews using the paper and literature before performing 18 pairwise comparisons, would take an average of 15 minutes. we recruited 11 researchers from our research group, ranging from phd students to associate professors, all with experience in reviewing and publishing at top ml/nlp conferences. each annotator dedicated 2.5 to 3 hours to evaluate 10 examples, performing 18 pairwise comparisons for each example. the study therefore focused on a dataset of 100 examples, with 10 being annotated twice to measure inter-annotator agreement. dominance scores were computed based on the results of the 1,980 undertaken comparisons. the interannotator agreement was computed using labels for 180 comparisons from 10 examples. to ensure consistency, detailed guidelines, criteria definitions, and examples were provided to annotators through the interface (examples in appendix f). ## conclusion in this study, we introduced the task of automated focused feedback generation for scientific writing assistance. it involves generating specific, actionable and coherent comments which identify weaknesses in a paper and/or suggest revisions. we presented swif 2 t , a system comprising planner, investigator, reviewer, and controller components. it is designed to gather context for a given paragraph and generate focused feedback. through automatic and human evaluations, swif 2 t demonstrated superiority over other llm-based approaches in similarity to human-written reviews, specificity, read-ing comprehension and overall helpfulness of its comments.