Focal Training and Tagger Decouple for Grammatical Error Correction (2023)

Abstract: In this paper, we investigate how to improve tagging-based Grammatical Error Correction models. We address two issues of current tagging-based approaches, label imbalance issue, and tagging entanglement issue. Then we propose to down-weight the loss of well-classified labels using Focal Loss and decouple the error detection layer from the label tagging layer through an extra self-attention-based matching module. Experiments over three latest Chinese Grammatical Error Correction datasets show that our proposed methods are effective. We further analyze choices of hyper-parameters for Focal Loss and inference tweaking.

## introduction grammatical error correction (gec) has been receiving increasing interest from the natural language processing community with the surging popularity of intelligent writing assistants like grammarly. in the english language, a series of benchmarks @xcite @xcite have been created for the evaluations of different methods. in many other languages, various emerging datasets with language-specific challenges are also attracting plenty of attention @xcite @xcite @xcite . existing methods for gec can be categorized into sequence-to-sequence approaches, tagging-based approaches, and hybrid approaches. sequence-to-sequence approaches require a larger amount of training data and usually rely on synthetic data for pretraining @xcite @xcite . tagging-based approaches adopt editing operations between the source text and the target text as training objectives @xcite . these methods are faster in inference and can achieve competitive performance as sequenceto-sequence approaches. hybrid models separate the tagging process and the insertion process into two stages, and can easily change word order with an extra pointer network module @xcite . in average cases, hybrid approaches can achieve sub-linear inference time. in this work, we are interested in tagging-based models due to their simplicity and high efficiency and would like to investigate how to further improve their performance. in the literature, there is also work on improving the performance of existing tagging-based models. for example, @xcite explored ensembles of recent transformer encoders in large configurations with various vocabulary sizes. for general-purpose improvements, existing methods range from optimizing training schemes to changing inference techniques. for training, @xcite explore how to enhance a model through generating valuable training instances and applying task-specific pretraining strategies. for inference, @xcite propose align-and-predict decoding (apd) to offer more flexibility for the precision-recall trade-off. from the perspective of system combination, @xcite propose a simple logistic regression algorithm to combine gec models effectively. different from the methods discussed above, we focus on improving tagging-based models from the perspective of model designing and learning. currently, gector @xcite is one of the representative tagging-based models. gector contains a pretrained transformer-based encoder with two linear classification layers as the tagger. one of the linear layers is used for label tagging, and the other for error detection. however, we identify the following issues with current tagging-based models: (1) label imbalance issue. the training labels contain a large portion of easy-to-learn labels and the distribution is highly transformer model 你 好 很 兴 知 高 认 ！ 你 [sep] skewed. existing learning methods use cross entropy with label smoothing as the loss function which is deemed as sub-optimal for this scenario. to solve the problems discussed above, we propose the following modifications to tagging-based models: (1) we use focal loss @xcite to counteract class imbalance and down-weight the loss assigned to correctly classified labels. (2) we decouple error detection from label tagging using extra attention matching module @xcite . we then verify the effectiveness of the proposed method over three recent chinese grammatical error correction datasets. through our experiments, we find that both focal loss and matching mechanisms contribute to performance gain. tagging scheme. we use a tagging scheme from gector @xcite with adapted vocabularies and operations by mucgec @xcite . specifically, the scheme computes an optimal token-level alignment between x and w. then for each aligned pair of tokens, there will be four choices for tagging labels: (1) keep for identical tokens, (2) delete if the token comes from x only, (3) replace_w if the token from x is replaced by the one from w, (4) append_w if the token comes from w only. for example, append_， and replace_识 in figure 1 . notice that if multiple insertions appear within one alignment, only the first one is used for training. the error detection labels are constructed from the tagging labels. it will be a correct label cor if tagged as keep else error label err. encoder. we use a transformer-based encoder to process the tokenized input text. we enclose the context with special tokens [cls] and [sep] and pass them into the bert model. we use the last layer of bert as the encoded hidden representation for the context. considering our tagging system is consistent with gector, we also use a mismatched encoder to get hidden representations of the original word, denoted as @xmath0, . . . , h n ). tagger. our tagger contains two separate linear classification heads. the label tagging head is conducted over the encoder's hidden representation h directly: the error detection head is decoupled from the label tagging head using an input constructed from h with a matching mechanism over its self-attended representation: training objective. in this paper, we choose focal loss to down-weight correctly classified labels: ## experiments we evaluate our method on three recent chinese grammar error correction datasets. ## limitations in this work, we have been focusing on improving the performance of tagging-based grammatical error correction. our work has the following limitations: (1) we work on three recent chinese grammatical error correction datasets. but there are many emerging datasets from various languages. we will add support for these languages on our github repository and make all resources publicly accessible. (2) we point out a limitation of inference tweaking, but it remains to be explored how to explain the phenomenon and derive better tweaking methods. ## conclusion in conclusion, focal training and tagger decoupling are effective in improving current tagging-based grammatical error correction models. however, it is also important to choose a suitable γ for focal loss considering the generalizability of the model. for the widely adopted post-processing technique inference tweaking, it depends on the model whether there will be significant performance gain.