A Report on FCG G en C hal 2022: Shared Task on Feedback Comment Generation for Language Learners (2023)

Abstract: We report on the results of the first ever shared task on feedback comment generation for language learners held as Generation Challenge (GenChal) in INLG 2022, which we call FCG GenChal. Feedback comment generation for language learners is a task where, given a text and a span, a system generates, for the span, an explanatory note that helps the writer (language learner) improve their writing skills. We show how well we can generate feedback comments with present techniques. We also shed light on the task properties and the difficulties in this task, with insights into the task including data development, evaluation, and comparisons of generation systems.

## introduction feedback comment generation for language learners is a task where, given a text and a span, a system generates, for the span, an explanatory note that helps the writer (language learners) improve their writing skills as exemplified in fig. 1 (for convenience, the task will be abbreviated as feedback comment generation, hereafter). in this regard, feedback comment generation is related to grammatical error detection and correction. in many cases, however, it is not enough to just point out an error with its correct form in order to help language learners with writing learning. instead, it is often essential for them to explain the underlying rules. in other words, it is essential in feedback comment generation to include more information than grammatical error detection and correction provide. we report on the results of the first ever shared task on feedback comment generation held as generation challenge (genchal) in inlg 2022, which we call fcg genchal. one of the goals of this report is to reveal how well we can generate feedback comments with present techniques. there is a wide variety of choices for generation methods that are applicable to this task. nevertheless, they have not yet been explored (at least, much less than in other generation tasks). another goal is to shed a light on the task properties and the difficulties in this task. specifically, we show, based on the results, insights into the task including data development, evaluation, and comparisons of generation systems. ## related work generally speaking, feedback comment generation is a task of text-to-text generation. the input text, which is written by a language learner, is transformed into another text explaining the writing rules. this implies that generation methods employed in other generation tasks such as machine translation (mt) may be effective in the present task. for example, feedback comments often refer to words and phrases appearing in the input text, and techniques for referring to words in the source text (e.g., copy mechanisms) will likely be beneficial. feedback comment generation has its own unique aspects. it should be emphasized that a feedback comment is generated against a span (of the input text or sentence) whereas only a text (e.g., a sentence or utterance) is dealt with in other major text-to-text generation tasks such as mt and dialog systems. in consequence, feedback comment generation systems have to output different texts for the exact same source sentence, depending on the ## data based on the work @xcite , we created two versions of new datasets for this generation challenge: feedback comments written in the same language as the target (input) text (i.e., english) and in a different language (specifically, japanese). the input texts (written by learners) are excerpts from the essays in ic-nale @xcite . we had experts, who had experience in english teaching, manually annotate all preposition errors in the input texts with feedback comments in english and japanese. after having finished all annotations, we looked into the results. it turned out that the overall quality of the obtained data was much higher in the japanese version than in the english. for this reason, we decided to use the japanese version in this fcg genchal; we translated the japanese feedback comments into english. overall, it took us approximately three years to create the final datasets. the results were split into training, development, and test sets. if a sentence contains more than one preposition error, it appears two or more times with different spans (in different lines). the split sets were provided for the participants, which are also available on the official fcg genchal web site. ## discussion as shown in sect. 5, all participating systems are based on a pre-trained, transformer-based generator while the baseline system uses a non-pretrained bilstm. this partially answers one of the research questions raised in sect. 2 (i.e., does a pre-trained general (native) language model work on learner writings?). the results show that pretraining and/or the architectures (likely both) contribute to performance improvement, although we need more investigation to confirm this argument. the top five systems use t5 as a generator while the rest use either gpt-2 or bart. the results prefer t5 as a generator for feedback comment generation. having said that, we need more investigations to confirm this argument, considering the amount of training, development, and test data. interestingly, some teams report that smaller models perform equal to, or even better than, the corresponding larger model (see their reports for the details). for example, they report that under the same condition, 't5-base' achieves a better bleubased f 1.0 than 't5-large' while bart-base' and 'bart-large' achieve a similar bleu-based f 1.0 . a possible reason for this is that the training set is not so large and that the amount is not enough to fine-tune a large model properly. here, one thing we should note is that these comparisons are based on bleu because manual evaluation was applied only to the final submission results (thus, one has to use automatic evaluation (e.g., bleu) to compare their system variants). manual evaluation may lead to a different conclusion. comparison between bleu and manual evaluation results provide an interesting insight into this task, which is summarized in fig. 3 . bleu and manual evaluation results correlate well (correlation coefficient: 0.85). however, the system rankings differ from those by manual evaluation when the difference in bleu is small. specifically, even if the difference is more than 0.01 (i.e., tmuued: 0.474 vs. nigula: 0.463), a reversal of a system ranking occurs. according to the obtained results, when the difference is larger than a certain value (e.g., 0.02 in this case), bleu might be a reliable measure to choose a better system (or a better method, or a better hyper-parameter setting). we need more investigations to confirm that this argument is correct. for the time-being, we do not have enough data to do so and we need manual evaluation to obtain reliable system rankings. at the same time, manual evaluation is costly and time-consuming. one of the necessary research directions is to explore more efficient ways of evaluation. another challenging direction is to pursue methods for @xmath0.e., not possible to generate a reliable feedback comment). considering practical use, it is important to decide not to generate when the system is not confident enough. in fcg genchal, four out of the seven systems are capable of @xmath1. their implementations are rather simple (e.g., simple rule-based) and their effects are limited; the difference between precision and recall is rather small as shown in ## conclusions in this paper, we have reported on the results of a new generation challenge called feedback comment generation for language learners. the bestperforming system achieves an f 1.0 of 0.62 in manual evaluation. the results suggest that pre-training and/or transformer-based methods are effective. they also suggest that smaller models within transformer-based methods perform better with the training data available. we have also reported insights into automatic and manual evaluation in feedback comment generation.