Grammatical Error Correction Systems for Automated Assessment: Are They Susceptible to Universal Adversarial Attacks? (2022)

Abstract: Grammatical error correction (GEC) systems are a useful tool for assessing a learner’s writing ability. These systems allow the grammatical proficiency of a candidate’s text to be assessed without requiring an examiner or teacher to read the text. A simple summary of a candidate’s ability can be measured by the total number of edits between the input text and the GEC system output: the fewer the edits the better the candidate. With advances in deep learning, GEC systems have become increasingly powerful and accurate. However, deep learning systems are susceptible to adversarial attacks, in which a small change at the input can cause large, undesired changes at the output. In the context of GEC for automated assessment, the aim of an attack can be to deceive the system into not correcting (concealing) grammatical errors to create the perception of higher language ability. An interesting aspect of adversarial attacks in this scenario is that the attack needs to be simple as it must be applied by, for example, a learner of English. The form of realistic attack examined in this work is appending the same phrase to each input sentence: a concatenative universal attack. The candidate only needs to learn a single attack phrase. State-of-the-art GEC systems are found to be susceptible to this form of simple attack, which transfers to different test sets as well as system architectures,

## introduction grammatical error correction (gec) systems can form a part of automated language fluency assessment: the number of edits from a candidate's input sentence to a gec system's grammatically corrected output sentence is indicative of a candidate's language ability, where fewer edits suggest better fluency. early gec systems were designed using hand-crafted rules @xcite , but since, data driven approaches, such as statistical machine translation @xcite , emerged. with encoder-decoder architectures dominating in neural machine translation, @xcite used recurrent neural networks @xcite to improve gec performance. now state of the art gec systems are based on the transformer @xcite architecture @xcite @xcite @xcite @xcite . despite the success of transformer-based deep learning systems, there is a shortcoming: @xcite discovered that neural networks are susceptible to adversarial attacks, where a small change at the input can yield large, undesired changes at the output of the model. in the gec setting, a candidate may seek to make a change to their input sentence, such that the system makes no corrections, resulting in zero edits between the source and prediction sequences, which falsely indicates perfect language fluency. given the highstakes of an assessment setting, it is particularly concerning if a candidate can engage in such malpractice. hence, this work explores the susceptibility of gec systems to adversarial attacks. gec systems operate on natural language inputs. in this domain, there are many proposed adversarial attacks @xcite , but on the whole they are inappropriate for sequenceto-sequence tasks, such as gec. @xcite ; @xcite ; @xcite ; @xcite introduced methods for adversarial attacks in sequence-to-sequence models. these works require multiple queries of the target system. however, a candidate cannot query a gec system. to solve this issue, this work uses a universal @xcite adversarial attack. here, the same universal attack phrase is appended to the end of all candidates' input sen-tences, i.e. a new candidate can simply acquire (e.g. through purchase) a fixed universal attack phrase to concatenate to their input and deceive a gec system used for automatic fluency assessment. this work also considers the transferability of a single attack phrase across different datasets and even architectures. further analysis is carried out to determine the aspects of gec systems that cause them to be susceptible to this form of attack. despite advances in natural language adversarial attacks, there has been less research on developing defence schemes. defence strategies can be categorized as model modification, where the model or data is altered at training time (e.g. adversarial training (yoo and qi, 2021)) or detection (raina and gales, 2022), where external systems or algorithms are applied to trained models to identify adversarial attacks. model modification approaches demand re-training of models and so detection approaches are preferred for deployed systems. note that for attacks on gec systems, detectors based on grammatical @xcite and spelling @xcite errors will fail. in this work, the most popular detection approaches: frequency guided word substitution @xcite ) (shown to outperform @xcite ); perplexity @xcite and model confidence @xcite ; are applied to detecting adversarial attacks on gec systems. ## related work in literature there has been limited work examining adversarial attacks for gec systems. however, some works have explored adversarial robustness. first, wang and zheng (2020) perform adversarial training to improve the performance of their gec system. their adversarial training scheme augments the training data with adversarial examples, generated through the insertion of common grammatical mistakes in grammatically correct sentences, where the insertions are tuned to exploit weak spots in the gec system. further, tang (2021) also seeks to increase robustness of gec systems in a post-training setting, through further training on adversarial examples generated from four different nlp adversarial attack schemes. these adversarial attack methods again are designed to fool the sequence-to-sequence gec system. finally, @xcite also augment the training data with adversarial examples, but focus on ensuring the adversarial examples mimic human grammatical errors by introducing noise at both a token level and embedding level. however, the above schemes are inappropriate for the attack setting in this work. first, the aim of the attack in this work is to perturb grammatically incorrect sentences to conceal grammatical errors. second, the existing works consider attacks specific to each input, whereas this work considers the more realistic setup of a universal adversarial attack. ## grammatical error correction grammatical error correction (gec) systems perform a sequence-to-sequence task, where an input word sequence, x 1:t , containing grammatical errors, is corrected for these errors by the system, with parameters, θ to predict the grammatically correct output word sequence, ŷ1:l , ŷ1:@xmath0 to evaluate the performance of a gec system, it is necessary to identify the edits made by the system and compare to the reference edits. an edit is defined as a modification (insertion, deletion or substitution) required on the input sequence x 1:t to make it match the target sequence, y 1:l . a popular edit extraction tool is errant @xcite , which uses a linguistically-enhanced alignment algorithm proposed by @xcite . edits between the input sequence, x 1:t , and hypothesised prediction sequence ŷ1:l can be found, ê1:p , ê1:@xmath1:t , ŷ1:l ). ## gec adversarial attack a targeted adversarial attack on an input text sequence, x 1:t aims to perturb it to generate an adversarial example x ′ 1:t ′ that ensures the output of a classifier, f(), is t, h() is some distance metric between the original and adversarial input sequences, ensuring the change is imperceptible. it is not simple to define an appropriate function h() for word sequences. perturbations can be measured at a character or word level. alternatively, the perturbation could be measured in the vector embedding space, using for example l p -norm based @xcite metrics or cosine similarity @xcite . however, constraints in the embedding space do not necessarily achieve imperceptibility in the original word sequence space. this work uses a simple variant of a levenshtein edit-based measurement @xcite which counts the number of changes between the original sequence, x 1:t and the adversarial sequence x ′ 1:t ′ , where a change is a swap/addition/deletion, and ensures it is smaller than a maximum number of changes, n . for a candidate planning to perturb their input sentence, the simplest attack is concatenation, where a fixed phrase is appended to their input @xcite @xcite , the aim of the adversarial attack on a gec system used for automated assessment, f() = s θ () (equation 4 ), is to maximally decrease the count of edits between the input sequence and the predicted sequence, i.e. a candidate wants to conceal their grammatical errors from the gec system. a single universal adversarial phrase, δ1:n is to be used for all candidates, i.e. once this universal phrase has been learnt from a set of j candidates, it can be sold to other candidates. hence, the cost function an adversary seeks to optimise is it is important to consider the interpretation of imperceptibility in the automated assessment setting. in many applications, measuring imperceptibility by counting number of added words, n , is inadequate as it can result in incomprehensible phrases that can easily be identified by a human reader. however, in this setting, there is no human reader, which demands the use of automated systems for identifying incomprehensible phrases. therefore, this work includes experiments to filter for adversarial attack words that do not compromise the integrity of an input sentence, when measured using a perplexity detector (introduced as a detection mechanism in section 5, equation 9 ) based on a state of the art language model. this ensures that an attack phrase remains imperceptible in an automated assessment setting. this work also investigates variations in the punctuation a candidate can use to concatenate an adversarial phrase to an input sentence. if '*' represents the form of punctuation, then to concatenate an adversarial phrase to the original phrase, we do: original phrase* adversarial phrase. ## defence for deployed systems, defence strategies that require re-training are undesirable. it is easier to use detection processes to identify and flag adversarial examples. this section considers how state of the art detection approaches can be applied to universal concatenation adversarial attacks on gec assessment systems, described in section 4. all detection approaches, d(), use a selected threshold, β to classify an input sequence, x 1:t as adversarial or not. when d(x 1:t ) > β, then the input sequence x 1:t is flagged as an adversarial example. to examine the performance of the detection process, this work uses precision-recall curves, where precision and recall values are calculated for a sweep over the threshold β. here, for each value of β, the precision and recall values are calculated (as in section 3), with adapted definitions for true-positive (number of samples correctly classified as adversarial), false-positive (number of samples incorrectly classified as adversarial) and false-negative (number of samples incorrectly classified as non-adversarial). a single-value summary is again obtained with the f 0.5 score, giving greater weighting to precision over recall, as it is more important to be correct in accusing candidates of mal-practice than finding all the candidates that cheat. the threshold with the highest f 0.5 score is selected for the detector d(). the recently dominating, frequency guided word substitution (fgws) @xcite algorithm is adapted for attacks on an assessment gec system. for the fgws algorithm, we generate a sequence x * 1:t from the original input sequence, x 1:t by substituting out low frequency words for higher frequency words. precisely, a subset of eligible words (for substitution) is found x @xmath2:t |ϕ(x) < γ}, where ϕ(x) gives frequency of word x and γ ∈ @xmath3. then, for each eligible word @xmath4, u(x) is found using synonyms. a replacement word x * is selected as x * = arg max @xmath5. hence, x * 1:t is generated by replacing each word x in x 1:t if ϕ(x * ) > ϕ(x). for the gec assessment system, s θ (), defined in equation 4, the fgws detection score is, smith and gal (2018) describe the use of uncertainty for adversarial attack detection, where adversarial samples are thought to result in greater epistemic uncertainty. in this work, negative confidence is selected as a simple measure of uncertainty. it is easiest to measure the confidence using the grammatically correct sequence output by the gec system, ŷ1:l (equation 1 ). the negative confidence detector score is calculated as, this works also explores the positive confidence detector, d pc (x 1:t ) = -d nc (x 1:t ). a final popular nlp detection approach is to consider the perplexity @xcite of the input sequence. it is expected that adversarial sequences have a greater perplexity than original samples. the perplexity detector, using some language model (lm), can be defined as, in recent years, grammatical error correction systems have been dominated by large (up to 11b parameters) transformer based architectures @xcite . using the f 0.5 metric defined in section 3, ## conclusions grammatical error correction (gec) systems can contribute to automated fluency assessment. the count of edits between a candidate's input and the grammatically correct output sequence from the gec system, is a measure of the candidate's ability in the language: fewer the number of edits, the better the candidate. however, this work showed that deep learning based gec systems are susceptible to adversarial attacks, where a candidate can cheat by adjusting their input sentence such that the predicted sequence from the gec system does not correct the existing grammatical errors. to model a realistic adversarial attack setting, this work restricts itself to a blackbox, universal attack approach, where the same adversarial phrase is appended to the end of all candidates' input sequences. this setting is particularly threatening because a candidate can cheat without querying the gec system even once -the candidate only has to acquire the attack phrase. it is found that the same universal attack phrase can be effective across multiple datasets and more interestingly can be transferred to deceive new, unseen architectures. this demonstrates that all gec systems have a worrying susceptibility to even the simplest attack forms.