Template-guided Grammatical Error Feedback Comment Generation (2023)

Abstract: Writing is an important element of language learning, and an increasing amount of learner writing is taking place in online environments. Teachers can provide valuable feedback by commenting on learner text. However, providing relevant feedback for every issue for every student can be time-consuming. To address this, we turn to the NLP subfield of feedback comment generation, the task of automatically generating explanatory notes for learner text with the goal of enhancing learning outcomes. However, freely-generated comments may mix multiple topics seen in the training data or even give misleading advice. In this thesis proposal, we seek to address these issues by categorizing comments and constraining the outputs of noisy classes. We describe an annotation scheme for feedback comment corpora using comment topics with a broader scope than existing typologies focused on error correction. We outline plans for experiments in grouping and clustering, replacing particularly diverse categories with modular templates, and comparing the generation results of using different linguistic features and model architectures with the original dataset versus the newly annotated one. This paper presents the first two years (the master’s component) of a research project for a five-year combined master’s and Ph.D program.

## introduction written corrective feedback on learner text is widespread in language education, and an active area of research in the field of second language acquisition (kang and @xcite . research has shown that properly administered teacher feedback has a positive effect on language acquisition @xcite , including in electronic settings @xcite . with the rise of shared online writing environments and e-learning platforms, it has become possible for teachers to assess and comment on learner text digitally. while these advancements in computerassisted language learning (call) are helping revolutionize language education, it remains true that writing frequent and context-appropriate feedback comments on essays is a time-consuming task for teachers. it would be beneficial to provide instructors with automatically generated suggestions when writing comments, allowing them to accept or edit suitable feedback comments and reject unsuitable ones. using similar technology, it is also possible to provide such feedback comments directly to learners in an intelligent tutoring setting as well. with such use cases in mind, we turn to the task of feedback comment generation. in nlp, feedback comment generation is the task of generating hints or explanatory notes for language learners @xcite . data consists of learner sentences, associated feedback comments, and offsets or spans to highlight where the comments were attached to the sentence. an example, taken from the icnale learner essays with feedback comments dataset described in @xcite foot_0 can be seen in figure 2 . this is one of a handful of corpora about this task, along with a translated subset used in genchal 2022 @xcite and a separate corpus developed by @xcite and expanded upon in @xcite . the commented icnale corpus is fairly small, as seen in additional challenges were revealed by @xcite and the participants of genchal 2022. first, generation is confounded by a many-to-one issue in which multiple comments which ultimately concern the same topic may use different wording. consider the following pair of sentences: *we reached to the station. because the verb "reach" is a transitive verb, the preposition "to" is not required. the targeted error is the same, but the comments are superficially different. this diversity can result in mixed generations which are less clear, as shown in @xcite . furthermore, there are a large number of very specific comments relating to particular words and their collocations. in relatively inflexible systems such as the neural retrieval model seen in @xcite , these are rarely output, since the same words would have to occur with the same errors to produce a match. in more flexible generation systems, such comments show a great deal of diversity and contribute to the mixed output problem. finally, generation systems can produce inaccurate or misleading comments which can lead learners astray, as reported by @xcite . it is important to constrain these false generations, which can have a negative learning effect or reduce confidence in the system. this research seeks to improve the generation of educationally effective feedback comments by addressing the above challenges. we outline plans to group feedback comments with a set of annotations which focus on the "topic" of each comment, based on its communicative purpose and its connection to an issue in the sentence when applicable. we identify highly variable or noisy feedback comment categories and replace such categories with modular templates. we also describe experiments with textual features and generation architectures to be used in testing the effects of the above approaches. it is hoped that these contributions can enable additional research into feedback comment generation for language learning. ## related work pedagogical feedback comments have long been studied in the field of education, including in the context of language learning. there is considerable debate about what kind of feedback works best and why, which includes dimensions such as directness @xcite , presence of metalinguistic terms @xcite , and hedging @xcite . while there are some detractors @xcite , written feedback has generally been found effective for language learning (kang and @xcite . turning towards the online environment of our task, we must consider systems which already exist. there are various tools for grammatical error correction (gec) and writing assistance, perhaps the most notable of which is grammarly foot_1 . we define the purpose of these tools as writing assistance, in which the goal is to improve the content of the document. this overlaps with, but is distinct from, the purpose of this work, which we define as learning assistance. our goal is to help learners notice and understand their errors, not just correct them. defining and suggesting changes in sentences is a necessary step in the process, but it is done with an eye towards a long term learning effect. the generation goal is therefore different. in our case, it is acceptable if we do not produce a comment for every error in the sentence, since we prioritize precision to avoid misleading students, and because a large number of overlapping and uncoordinated feedback comments can overwhelm and demotivate students @xcite . on the other hand, a gec or writing support system like grammarly ideally has something to offer for all issues in a sentence. we also place more emphasis on explaining what is wrong and particularly why, rather than what specific edits should be made, which the comments in this task often hint at rather than provide outright. this would quickly become frustrating in a writing support environment, but in the context of education, such comments have been found to be effective for long-term learning @xcite . in the context of natural language processing, feedback generation was first formally defined in @xcite , followed by the release of a dataset containing learner sentences and feedback comments, constructed from essays from the international corpus network of asian learners of english (icnale) @xcite . the dataset contains triples consisting of english sentences written by learners, feedback comments added by professionals, and the offsets designating the target span of the comment. there were also developments by @xcite , who created a system which constructed feedback templates from word collocations, and @xcite , who added additional annotations to a corpus of textual revisions @xcite and investigated the revision outcomes of various kinds of feedback comments. @xcite performed experiments on the icnale feedback dataset, revealing challenges faced by models using neural retrieval, simple generation, and retrieve and edit @xcite architectures. namely, the retrieval model can not generalize, the retrieve-and-edit model over-edited in an unconstrained manner, and the simple generation and retrieve-and edit models both produced mixed or misleading outputs. following that, there was a shared task on feedback comment generation in genchal 2022. teams demonstrated various modeling and preprocessing techniques, particularly that it is possible to extract detailed linguistic features from the sentences and comments using existing nlp tools such as parsers and gec systems and use them to enhance feed-back comment output. 3 we discuss several such options in section 3.4. ## research plan based on the above literature, we have identified two major challenges in this task: the preposition "for" cannot be followed by the base form of a verb. use a to-infinitive instead. the preposition "for" indicating the purpose of the sentence are followed by nouns including gerunds. you cannot put a verb in its original form after a preposition. use to-infinitive to indicate the purpose. the learner sentences have little in common beyond the presence of "for + base form of a verb," and the feedback comments are different in length and detail. the latter contains several words such as "gerunds" and "purpose" which the former lacks. there is also a difference in terminology: "base form" vs. "original form." it is possible that these differences may cause the two comments to be treated somewhat differently by models. it is ideal if they can be assigned to the same group. ## future directions this paper presents details about the first two years (the master's component) of a research project for a five-year combined master's and ph.d program. there are many additional research directions and concepts which can be incorporated before the final thesis is complete. for generation, given the mixed output issue observed in previous studies, and the overlapping nature of the tagging system, it may be prudent to separate generation models. different models could be used for abstract comments versus the operational and grammatical pattern ones. if a more abstract comment is generated, the grammatical figure 5 : example of a feedback comment rewritten as a template and simplified. this example template has two slots to fill with citations, a "problem word" whose mistaken combination caused the error ("agree"), and a "reference word" for indicating a position for a suggested operation ("it"). there is also a general lack of corpora for the task of feedback comment generation. given that each teacher has their own idiosyncrasies in correcting learner text, it is highly desirable to collect more data from a variety of writers. furthermore, the icnale learner essays with feedback comments dataset contains only essays written by learners from china, india, japan, korea, thailand, and taiwan, and the number of comment writers is limited. the learner's cefr levels range from a2 to b2, the ages range from 15 to 37, and all writing is in the context of a single-draft essay. working with corpora with different learner first languages, age groups, or writing tasks may further affect the annotation sets and clusters discussed in this work, as well as provide valuable training data in the form of new and unique sentence-comment pairs, particularly for categories such as language transfer. we have preliminary plans to construct a new corpus of learner sentences, feedback comments, and comment topic labels, which will be informed by the insights gained during this research. ## summary to assist in the task of feedback comment generation, we add manual labels to the feedbackenhanced icnale dataset which consider broadscope errors, explore grouping comments using these manual labels as a reference, craft modular templates for highly diverse categories of feedback comments, and perform modeling experiments with a variety of architectures and using features obtained by parsers and gec tools, reporting on the best combinations.