The T okyo Tech and AIST System at the G en C hal 2022 Shared Task on Feedback Comment Generation (2023)

Abstract: This paper describes the Tokyo Tech and AIST system in the GenChal 2022 shared task, which is the first shared task of feedback comment generation. We adopted five methods: data cleaning, fine-tuning pre-trained models, correcting errors in learners’ sentences, appending a correcting operation, and filtering out irrelevant outputs. Our system achieved F1 = 43.4 on the test dataset.

## introduction recently, @xcite proposed a novel task called feedback comment generation (fcg), wherein feedback is provided to help writers improve their skills, especially in the context of computer-assisted language learning. the input of an fcg model is a learner's sentence, and the output is a comment given as feedback to the learner. the genchal 2022 shared task is the first shared task of the fcg task. this paper describes the system developed in our study, which encompasses the following five methods: this paper is organized as follows. section 2 describes the shared task and its dataset. section 3 details the methods outlined above. section 4 presents the experimental setup. section 5 shows the results. section 6 concludes this paper. all of our code has been publicly released for reproducibility foot_0 . ## task and dataset description the genchal 2022 shared task was proposed by @xcite to address fcg. the organizers released a new dataset for this task, wherein original texts written by english learners were borrowed from icnale @xcite . the input of this task is a pair consisting of the learners' text and a span indicating the feedback comment's location. the input text is written in english and tokenized. the span is provided as input in this task, although it can be detected by grammatical error detection models. for example, the sentence "it is a problem for health ." has an error, which we can correct by replacing for to of. the span is character-level and colon-separated, and the position indicates a 0-indexed point between characters including whitespace. therefore, the span of this example is 16:19, wherein the start position is 16 and the end position is 19. the output of this task is feedback comment, which must be informative beyond merely an indication of 'correct' or 'incorrect'. furthermore, specific words and phrases in feedback comments are annotated using brackets. grammar terms and idiomatic patterns are bracketed @xmath0, whereas quotations from the learner's sentence are bracketed @xmath1. miscellaneous quotations and words or phrases to highlight can be annotated using '' ''. for example, the feedback comment for the sentence in the last paragraph can be: @xmath2. simply use ''of'' in this case. the special @xmath3. in this task, feedback comments are written in english, while another choice is using the learner's native language as in @xcite . the released dataset was split into training, development, and test subsets, which contain 4868, 170, and 215 sentence pairs, respectively. feedback comments in the test dataset were not released during the shared task period. the bleu @xcite score is adopted as the automatic evaluation metric. to incorporate the use @xmath4, task submissions are evaluated by bleubased f 1 score. the precision (p ), recall (r) and f 1 score are calculated as follows: ## experimental setup we used the dataset released by the shared task and selected the best epoch for each training trial using the validation dataset. we fine-tuned models for 100 epochs with saving checkpoints at five-epoch intervals. we used the cross-entropy loss, adamw optimizer @xcite with a weight decay of 0.01 and gradient clipping of 1.0, and a learning rate of 0.0001 with a constant scheduler. we employed bart small and gpt-2 small to compare both models and gpt-2 large for the final submission. we used nvidia a100 gpu with 40 gib memory for all experiments and varied the batch size for model size to ensure an efficient use of the gpu memory. we set the maximum tokens per batch to 2,000 for bart/gpt-2 small and 250 for gpt-2 large, and accumulated every four batches for bart/gpt-2 small and 32 for gpt-2 large, thus setting the number of maximum tokens for each step to 8,000. ## conclusion in this paper, we described our system for the gen-chal 2022 shared task. we employed five methods: data cleaning, fine-tuning pre-trained models, correcting errors in learners' sentences, appending a correcting operation, and filtering. we fine-tuned bart and gpt-2 and then selected gpt-2 for submission. we verified that @xmath5. our final submission was obtained using gpt-2 large with appending a correcting operation and filtering without correcting non-target errors. our