Shared Task on Feedback Comment Generation for Language Learners (2021)

Abstract: In this paper, we propose a generation challenge called Feedback comment generation for language learners. It is a task where given a text and a span, a system generates, for the span, an explanatory note that helps the writer (language learner) improve their writing skills. The motivations for this challenge are: (i) practically, it will be beneficial for both language learners and teachers if a computer-assisted language learning system can provide feedback comments just as human teachers do; (ii) theoretically, feedback comment generation for language learners has a mixed aspect of other generation tasks together with its unique features and it will be interesting to explore what kind of generation technique is effective against what kind of writing rule. To this end, we have created a dataset and developed baseline systems to estimate baseline performance. With these preparations, we propose a generation challenge of feedback comment generation.

## introduction feedback comment generation for language learners is a task where given a text and a span, a system generates, for the span, an explanatory note that helps the writer (language learner) improve their writing skills (for convenience of explanation, the task will be abbreviated as feedback comment generation, hereafter). the target language(s) can be any language, but we limit ourselves to english input texts and english feedback comments in this challenge. as shown in figure 1 , a feedback comment is typically made about erroneous, unnatural, or problematic words in a given text so that the writer can understand why the present form is not good together with the underlying rule. in this regard, feedback comment generation is related to grammatical error detection and correction. in many cases, however, it is not enough to just point out an error with its correct form in order to help language learners with writing learning. instead, it is often essential for them to explain underlying rules, which makes the task different from the conventional grammatical error detection/correction. in other words, it is essential in feedback comment generation to include more information than grammatical error detection/correction provide. at the same time, unconstrained generation would make the task infeasible in terms of system development and evaluation. with this in mind, we set some constrains to the task to be explored in this generation challenge as described in section 2. for example, the input is limited to a sentence (and a span) instead of a text. the motivations for this challenge are as follows. a practical motivation is already mentioned above. it will be useful if a computer-assisted language learning system can provide feedback comments just as human teachers do. theoretically, feedback comment generation has a mixed aspect of other generation tasks together with its unique features as described in section 3. it will be interesting to explore what kind of technique is effective against what kind of writing rule. one of the goals of this challenge is to reveal how well we can generate feedback comments with existing techniques. there is a wide variety of choices as generation methods that are applicable ## related work generally speaking, feedback comment generation is a task of text-to-text generation. it then can be abstractly regarded as a machine translation (mt) problem where the input text, which is written by a language learner, is translated into another text explaining writing rules. this implies that generation methods employed in mt or other related research areas may be effective in the present task. for example, feedback comments often refer to words and phrases appearing in the input text, and techniques for referring to words in the source text (e.g., copy mechanisms) will likely be beneficial. unlike mt, the equivalence between the source and target texts in meaning do not hold. instead, the target text is a feedback comment that explains why the source is not good together with the underlying rule. from this point of view, the present task is related to research in dialogue systems. feedback comment generation has its unique aspects as well. it should be emphasized that a feedback comment is generated against a span (of the input text or sentence) whereas only a text (e.g., a sentence or utterance) is dealt with in other major text-to-text generation tasks such as mt and dialog systems. in consequence, feedback comment generation systems have to output different texts for the exact same source sentence, depending on given spans. the source and target languages are also unique. in this challenge, both are english, but there is room for discussion whether they fall into the same language class. the former is learner english, and inevitably it contains erroneous/unnatural words. even within correct sentences, grammar, expressions, and style are expected to be used differently from canonical english. this brings out further research questions related to the source and target languages. for example, which is the best setting of vocabularies -only one common vocabulary for the source and target, or one for each? does a pre-trained general (or native) language model work well to model learner english? there are a number of unaddressed research questions like these. feedback comment generation is also related to grammatical error detection/correction. the stateof-the-art methods typically solve the problems as sequence labeling (e.g., @xcite ) or mt with dnns (e.g., junczys-dowmunt et al. (2018); napoles and callison-burch (2017); @xcite ). recently, a dnn-based sequence labeling method is combined with symbolic transformations @xcite , which can be a good source of information to generate feedback comments. some researchers @xcite @xcite made an attempt to develop rule-based methods for diagnosing errors in line with grammatical error correction. researchers started to apply more modern techniques. @xcite showed that a neuralretrieval-based method was effective in preposition feedback comment generation. @xcite proposed a method that used grammatical error correction and templates to generate detailed comments. @xcite and gkatzia et al. (2014) proposed methods for automatically choosing feedback templates based on learning history. the availability of datasets for research in feedback comment generation has been increasing. nagata (2019) released a dataset consisting of feed-back comments on preposition use. they marked up erroneous prepositions and annotated them with feedback comments. nagata et al. (2020) extended it to other grammatical errors and also other writing items such as discourse and lexical choice. pilan et al. (2020) released a unique dataset where feedback comments on linking words were annotated. ## preparation based on the work @xcite , we created a new dataset for this generation challenge. the original texts are excerpts from the essays (written by learners of english) in ic-nale @xcite . we had native speakers of english, who had experience in english teaching, manually annotated all preposition errors with feedback comments in english. the dataset will be split into training, development, and test sets. note that training and development sets consist of the whole essays, meaning that they contain all sentences no matter whether they contain feedback comments or not (i.e., error free essays are included in the sets). also note that a sentence can be annotated with more than one feedback comment. in contrast, the test set only contains sentences with exactly one feedback comment each as described in subsection 2.2. if a sentence contains more than one preposition error, it appears two or more times with different spans (in different lines). they will be provided for the participants in due course. we also developed a web-based submission system that takes system outputs the participants submit. the system checks the output format of the submission and calculate its score (i.e., bleu). we also implemented two baseline systems for this challenge. one is a deep neural network (dnn)-based retrieval system that retrieves the most similar instance to the input sentence and outputs it as a generation result. the other is a text generation system based on a dnn encoderdecoder with a copy mechanism. as a pilot study, we tested them on the dataset @xcite . for manual evaluation, we trained a professional annotator who had more than ten years of experience in english syntactic annotation and two years of experience in professional english writing teaching. the person and the first and third authors independently evaluated the generation results. feedback comment as either appropriate or not, following the manner described in subsection 2.2. as a result, it turned out that the retrieval system and the text generation system achieved an f 1 of 0.35 and 0.43, respectively foot_0 . inter-evaluator agreements (cohen's kappa coefficient) were considerably high, ranging from 0.86 to 0.92. these results imply that the present task is not easy one, but also not completely insolvable. ## organizers the organizing group comprises six people as shown in the authors of this paper. all members have engaged in natural language research related to language learning and education for many years and some of them have organized several workshops and shared tasks in the domain. the first author developed the dataset. the second author developed the submission system together with a web page for this challenge. the two mainly designed this generation challenge with help from the other members. the third author implemented the baseline systems with the first author. they were also involved in the pilot manual evaluation. all will be involved in the actual generation challenge including evaluation and paper publication. although the trained professional evaluator is not included in the organizing group, the person will play a major role in manual evaluation. ## conclusions in this paper, we have described a new generation challenge called feedback comment generation for language learners. we have explored the task, describing the task definition, the related work, and the dataset to be used.