Character Transformations for Non-Autoregressive GEC Tagging (2021)

Abstract: We propose a character-based non-autoregressive GEC approach, with automatically generated character transformations. Recently, per-word classification of correction edits has proven an efficient, parallelizable alternative to current encoder-decoder GEC systems. We show that word replacement edits may be suboptimal and lead to explosion of rules for spelling, diacritization and errors in morphologically rich languages, and propose a method for generating character transformations from GEC corpus. Finally, we train character transformation models for Czech, German and Russian, reaching solid results and dramatic speedup compared to autoregressive systems. The source code is released at https://github.com/ufal/wnut2021_character_transformations_gec .

## introduction the current state of the art for grammatical error correction (gec) is achieved with encoder-decoder architectures, leveraging large models with enormous computational demands @xcite . as such autoregressive approach is slow on inference and is impossible to parallelize, it has recently been suggested to replace autoregressive sequence-to-sequence decoding with per-token tagging to enable parallel decoding, achieving a dramatic speedup by a factor of 10 in nmt @xcite and very recently, also in gec @xcite . @xcite approaches gec as a tagging task, discriminating between a set of wordlevel transformations. the designed set is efficient for english corpora, which rarely contain spelling errors, and for english language, which does not have diacritization marks and its morphology is very modest compared to morphologically rich languages such as czech or russian. using a set of word-level transformations designed for english, all character-level corrections would have to be handled by generic word-for-word replace rule, leading to an explosion of rules. we therefore suggest character transformations on subword level. moreover, our transformations are automatically inferred from the corpus as opposed to being manually designed. our approach has the following advantages: we present an oracle analysis of various transformations sets at different levels, in english and three other languages: czech, german and russian. we find that the word-level set of rules may be suboptimal for morphologically rich languages and corpora with spelling errors and diacritics. finally, we train models with character transformations for non-autoregressive grammatical error correction in czech, german and russian, reaching solid results and dramatic speedup compared to autoregressive systems. algorithm 1: extended lcs alignment input: input subwords s, gold characters g w[:, :] ← 0 for i ← |s| to 1 do for j ← |g| to 1 do w[i, j] ← w[i + 1, j] for l ← 1 to max(|g| -j, 8 + 3|s[i]|) do g ← g[j : j + l] if g.isspace() then continue c ←      1 if s[i] = g 0.75 if s[i].strip() = g.strip() 0.5 • levenshsimilarity(s[i], g) w[i, j] ← max(w[i, j], c+w[i, j +1]) end end end return alignment with weight w[0, 0], as in lcs with an attempt at non-autoregressive grammatical error correction for languages other than english, with character transformations applied at subwords, inferred automatically from parallel gec corpus. ## transformations given that we encode an input sentence using bert @xcite , it is natural to represent it using a sequence of subwords. we prepend a space to every first subword in a word and use no special marker for other subwords. note that the subwords might not correspond directly to parts of input, because the bert uncased model strip input casing and diacritics. ## transformations upper-bound f-score to assess the effect of number and type of transformations, we compute the potential maximum f 0.5 score with the maxmatch m 2 scorer @xcite . we generate transformation dictionary from the training portion of the following gec corpora: conll-2014 shared task for english @xcite , akces-gec @xcite for czech, falko-merlin gec @xcite for german and rulec-gec @xcite for russian. the sizes of the datasets are quantified in we also generate transformations from synthetic data augmentation used for training (section 5); to prevent the explosion of the transformation dictionary, we consider only 1000 synthetic sentences, except for russian, which employs 5000 synthetic sentences because of very small authentic data. finally, we add a special uncorrectable error dictionary element, indicating an error that cannot be corrected by any dictionary transformation. to encode a gold correction with a transformation, we first try looking it up in the dictionary. if it is not present, we go through all dictionary transformations in random order, accepting the first one producing the correct output. if no transformation match, we resort to the uncorrectable error (during prediction, it will be replaced by the input token). we show all combinations (character/string at words/subwords), using cased and uncased mbert, in figure 2 . clearly, character transforma- language dataset sentences czech akces-gec 47 371 german falko-merlin 24 077 russian rulec-gec 12 480 ## experiments we train the character subword gec tagging model using the char-at-subword transformation, which have achieved the best upper-bound score. 1 we train the character subword gec tagging model (char-at-subword) for czech, german and russian, in two stages: first, models are trained on a large synthetic corpus, generated by a reimplementation of @xcite . then, the models are finetuned on a mixture of synthetic and authentic data in ratio 1:2. the authentic data used in the second stage are akces-gec @xcite for czech, falko-merlin gec @xcite for german and rulec-gec @xcite for russian. the model is based on a pretrained bert encoder @xcite , specifically bert-basemultilingual (uncased for czech and german, cased for russian). after encoding the tokens, we add a simple softmax classifier that projects embeddings for each subword into a distribution over a set of transformations (section 3) generated from authentic data with a limited addition from synthetic data (section 4). we generate 7.7k transformations for czech, 4.3k transformations for german and 3.1k transformations for russian. gec models based on transformer and bert-encoder were shown to perform better when applied iteratively @xcite ; circled numbers 1 , 2 and 3 denote that we kept transformations present at least once, twice or three times in the training data, respectively (larger means less transformations). @xcite . therefore, we experiment with multiple iterations and report results both for single iteration and four iteration phases after which we did not observe significant improvements. we train both the fully-connected network and bert with adamw optimizer @xcite which minimizes the negative loglikelihood. both for pretraining and finetuning, the learning rate linearly increases from 0 to 5 • 10 -5 over the first 10000 steps and linearly decreases to 0 over 20 epochs. we use the batch size of 2048 sentences and clip each training sentence to 128 tokens. we pretrain each model for circa 14 days and finetune it for circa 2 days on nvidia p5000 gpu and select the best checkpoint according to development set. we experimented with weighting all classes different from the keep instruction by a factor of 3. it turned out effective only for pretraining russian. we present the results of our models in ## conclusion and future work we proposed a character-based method to generate target transformation instructions for gec tagging models, as an alternative to autoregressive models. we compared the character transformations to previously used word-level transformation instructions and have shown that characterbased rules have better coverage and scale better in czech, german and russian. moreover, we trained character-based gec tagging models for these languages. the source code is available at https://github.com/ufal/wnut2021_ character_transformations_gec . for future work, we propose to investigate ways to generate synthetic data to achieve better coverage of the target transformation set, since the current process for generating synthetic errors is well suited for encoder-decoder models, but may fail to cover certain transformations.