R ed P en N et for Grammatical Error Correction: Outputs to Tokens, Attentions to Spans (2023)

Abstract: The text editing tasks, including sentence fusion, sentence splitting and rephrasing, text simplification, and Grammatical Error Correction (GEC), share a common trait of dealing with highly similar input and output sequences. This area of research lies at the intersection of two well-established fields: (i) fully autoregressive sequence-to-sequence approaches commonly used in tasks like Neural Machine Translation (NMT) and (ii) sequence tagging techniques commonly used to address tasks such as Part-of-speech tagging, Named-entity recognition (NER), and similar. In the pursuit of a balanced architecture, researchers have come up with numerous imaginative and unconventional solutions, which we’re discussing in the Related Works section. Our approach to addressing text editing tasks is called RedPenNet and is aimed at reducing architectural and parametric redundancies presented in specific Sequence-To-Edits models, preserving their semi-autoregressive advantages. Our models achieve F0.5 scores of 77.60 on the BEA-2019 (test), which can be considered as state-of-the-art the only exception for system combination (Qorib et al., 2022) and 67.71 on the UAGEC+Fluency (test) benchmarks. This research is being conducted in the context of the UNLP 2023 workshop, where it will be presented as a paper for the Shared Task in Grammatical Error Correction (GEC) for Ukrainian. This study aims to apply the RedPenNet approach to address the GEC problem in the Ukrainian language.

## introduction the gec challenge has been tackled with various techniques, including the traditional autoregressive (ar) neural machine translation (nmt) using the transformer architecture @xcite , as well as additional methods that we refer to collectively as inference optimized (io). the existing io methods for gec can be broadly categorized into two groups, as described further. the first group is non-autoregressive feed forward (ff) approaches which involve a single forward pass -through the model and provides tokenlevel edit operations, such as the approach proposed in @xcite , @xcite . the advantage of ff approaches is their fast inference speed. however, their limitations lie in how they maintain consistency between interrelated edits, which leads to the need for iterative sentence correction approaches. the iterative sentence correction process solves some issues with interrelated corrections. however, it introduces new challenges. the absence of information about the initial input state could potentially lead to substantial modifications of the text meaning and structure, including rewording, word rearrangement, and the addition or removal of sentence components. the second category consists of inference optimized autoregressive (ioar) models, which can be further separated into two subcategories: (i) sequence-to-edits (seqtoedits). this category encompasses works such as @xcite , @xcite , @xcite , and the redpennet model examined in this paper; (ii) the recently proposed input-guided aggressive decoding (igad) approach @xcite , which has been proven effective for gec tasks, as demonstrated in the study @xcite . more information about these model categories can be found in the related work 4 section. in our study, we propose redpennet, which is an ioar model of the seqtoedits subtype. red-pennet utilizes a single shallow decoder @xcite for generating both replacement tokens and spans. during the generation of edit tokens, the encoder-decoder attention weights are used to determine the edit spans. for these attentions, pre-softmax logits are fed as inputs to a linear transformation which predicts the position of the edit in the source sentence. this approach is similar to the method described in pointer networks @xcite . additionally, we train compact taskspecific decoder bpe vocabularies to reduce the cost of the pre-softmax dot operation, making it more efficient for predicting replacement tokens. the redpennet model is also capable of tackling the challenge of multilingual gec @xcite . to achieve this, specialized shallow decoders need to be trained for different languages. this gives the ability to use a single model with a multilingual pre-trained encoder and languagespecific decoders. our proposed solution has a design that enables converting the input sequence into any output sequence, achieving competitive results in solving the gec task. ## training decoder bpe vocabularies in the traditional implementation of the transformer model, a shared source-target vocabulary is utilized for both the decoder and the encoder, as described in @xcite . it is evident that the pre-softmax linear transformation required to transform the decoder output into predicted nexttoken probabilities is computationally expensive. its computational complexity can be expressed as o(d • v), where d is the output dimension of the model and v is the decoder vocabulary size. if the gec task is approached by generating correction strings for the edits and using autoregressive decoding for this purpose, we tend to think that the information entropy of the generated sequences will be significantly lower compared to that of the input sequences. our belief is based on the following two assumptions: therefore, a smaller bpe vocabulary will be sufficient to create efficient representations of sequences of corrections. in section 5.1.2, we test this hypothesis on one of the languages, as the example shows. ## related work in the context of the gec task, the closest family of approaches to redpennet is the autoregressive approaches, specifically the seqtoedits subtype. they include models such as lasertagger, erroneous span detection and correction (esd&esc), and seq2edits comparison with which is important for understanding the impact of our work. these models share the advantage that the number of autoregressive steps are based on the number of necessary edits to the original text, rather than the length of the input text. we will evaluate each of these approaches to the gec problem in this section. in this section, we will also discuss the aggressive decoding approach, which has evolved from the traditional sequence-to-sequence approach. the seq2edits (stahlberg and kumar, 2020) approach predicts a sequence of n edit operations autoregressively from left to the right. each edit operation is represented as a 3-tuple (tag, span, token) that specifies the action of replacing. the approach allows constructing an edit sequence for any pair (x, y). tag prediction also improves explainability in the gec task. for 3-tuple generation, a divided transformer decoder is used, and the tag and span predictions are located between its parts. seq2edits approach is similar to redpen-net in the following (i) generation of spans and replacement tokens within the same autoregressive step, (ii) using pointer networks to predict spans. the difference between compared approaches is: 1. redpennet uses a single decoder stack to generate tokens and spans. 2. the seq2edits approach is different in terms of generating multi-token edits. according to @xcite , an edit that has at least two tokens (multi-token edit) represents 10% of all edits from the conll-2014 test set. as mentioned before, in the seq2edits approach, each step of the autoregressive process predicts an edit which consists of a 3-tuple (tag, end span, replacement token), where the replacement token is a sub--word. according to the cited articles @xcite , the seq2edits approach has the capability of representing multi-token edits as a list of single-token edits, where the tag and span remain unchanged, with only the replacement token being changed. in terms of the redpennet approach, a single edit can be represented by multiple autoregressive steps, allowing a more natural generation of multi-token edits. 3. in the redpen-net approach, decoder-specific positional encodings are added to the decoder inputs at the bottom of the decoder stack. this allows the model to effectively utilize the order of multi-token edits. the approach presented in seq2edits does not clearly state the location in the divided transformer decoder, where positional encodings can be utilized. the absence of such encodings can result in difficulty for the model in comprehending the order of the replacement tokens being inserted within the same span positions. 4. redpennet uses a pre-calculated, task-specific version of the bpe decoder vocabulary to generate edit tokens, thus reducing the cost of the pre-softmax linear transformation. the lasertagger @xcite approach deploys an autoregressive transformer decoder to annotate the input sequence with tags from precalculated output vocabulary. with the limited size of the tags, vocabulary minimizes the cost of the pre-softmax linear transformation, making lasertagger the fastest approach among the ioar seqtoedits architectures. however, the redpen-net model presents several key differences: (i) it uses a bpe vocabulary instead of a tag vocabulary, (ii) it generates edits rather than tagging the input sequence, and (iii) it can produce a sequence of tokens for each edit. in the esd&esc @xcite approach, the task of solving the gec editing problem is divided into two subtasks: erroneous span detection (esd), where incorrect spans are identified through binary sequence tagging, and erroneous span correction (esc), where the correction of these spans is performed using a classic autoregressive approach that implies generation of edits for tokens surrounded by annotated span tokens. redpennet shares some similarities with this architecture, as it also utilizes autoregressive generation of a sequence of edit tokens, separated by control tokens that are part of the decoder vocabulary. the esd&esc approach differs from redpennet in several key aspects. 1. firstly, redpennet predicts the span positions in a one-by-one manner at the decoder level, while the esd&esc approach uses a separate encoder to generate spans. however, the esd approach has the same limitations as the ff family, since the esd tags may not always be consistent, leading to difficulties in maintaining consistency between interrelated edits. the esc decoder during generation will not have the capability to fully rectify the situation, as it will be confined to the range of the annotated span tokens. 2. redpennet approach is capable of decomposing neighboring errors in the input text into multiple edit operations, if necessary. conversely, the esd approach merges nearby errors in a single span. the aggressive decoding @xcite ) method accelerates the ar calculations for the task by utilizing the input tokens as drafted decoded tokens and autoregressively predicting only those portions that do not match. this leads to a significant improvement in inference speed. the disadvantage of the igad approach is that it requires the use of a shared vocabulary with the encoder during decoding. therefore, even when the input and output sequences are the same, igad requires a significant number of floating point operations for the pre-softmax linear transformation in the decoder which is calculated using the formula: o(v • d • l), where v is the vocabulary size, d is the model depth, and l is the input length. this problem becomes more obvious in the case of using pre-trained multilanguage models, which traditionally have larger encoder vocabularies and corresponding matrix embeddings. the impact of decoder vocabulary is analyzed in section 3. additionally, since the length of the output sequence in igad is directly tied to the length of the input sequence, the issue of quadratic complexity in attention mechanisms remains in the decoder. this can be a challenge when dealing with long sequences and requires the use of specialized transformer architectures in the decoder. it is worth mentioning, that the highlight and decode technique described in our previous study @xcite . similar to the erroneous span detection (esd) component in the esd&esc approach, a binary sequence tagging model was used to identify incorrect spans. subsequently, a broadcast binary sequence mask was element-wise multiplied to a special "highlight" embedding. the result of this operation was added to the encoder output at the bottom of the decoder stack. this allowed the decoder to predict the replacement tokens only for the "highlighted" spans. however, as outlined in the mentioned article, this approach had a list of limitations. ## conclusion while there has been a significant amount of research in the field and many tailored architectures have been proposed, a universally accepted neural architecture for text editing tasks that involves highly similar input and output sequences has yet to be established. this has prevented the creation of an industry standard that can be included in default toolkits for popular machine learning libraries and mlops tools. our proposed redpennet is an attempt to create a universal neural architecture that is not overloaded with design nuances and is capable of implementing any source-to-target transformation using a minimal number of autoregressive steps. the redpennet architecture is a classic transformer, and the only differences lie in how we form decoder input embeddings and interpret outputs and attention scores.