Don’t take “nswvtnvakgxpm” for an answer –The surprising vulnerability of automatic content scoring systems to adversarial input (2020)

Abstract: Automatic content scoring systems are widely used on short answer tasks to save human effort. However, the use of these systems can invite cheating strategies, such as students writing irrelevant answers in the hopes of gaining at least partial credit. We generate adversarial answers for benchmark content scoring datasets based on different methods of increasing sophistication and show that even simple methods lead to a surprising decrease in content scoring performance. As an extreme example, up to 60% of adversarial answers generated from random shuffling of words in real answers are accepted by a state-of-the-art scoring system. In addition to analyzing the vulnerabilities of content scoring systems, we examine countermeasures such as adversarial training and show that these measures improve system robustness against adversarial answers considerably but do not suffice to completely solve the problem.

## introduction automatic content scoring @xcite is an educational task where free-text answers to a content question are automatically scored based on the conceptual correctness of the answers. consider for example the question what is the favorite food of pandas? with an expected answer such as pandas eat bamboo. a content scoring system which only accepted this exact same answer would be very limited, as there are many other ways to express the same fact, e.g. pandas like to eat bamboo, pandas feed on bamboo, or simply bamboo. in addition, students might make all kinds of spelling and grammatical mistakes, which are usually ignored in content scoring as long as they do not interfere with understandability of the answer. thus, pendas are eating bambboo would still be acceptable. a content scoring system must therefore be able to generalize beyond a certain set of a prior known correct answers, e.g. by taking variance in spelling @xcite and from other sources into account @xcite . for example, many supervised scoring systems rely on character-level n-gram features @xcite . without those features, the system would not be able to detect that bambboo is likely to mean bamboo as they both e.g. contain the trigrams 'bam' and 'amb'. in fact, such systems show remarkable resilience against spelling errors, where even introducing up to 50% spelling errors in each answer does not significantly influence scoring performance @xcite . this ability of a system to generalize beyond previously seen examples makes it potentially vulnerable to adversarial behaviour. in the educational domain such behaviour has long been known under the term cheating. when students have no idea about a task, they could, for example, try to write an answer combining several potential answers, such as banana bamboo carrots grass, in the hope to accidentally include the right one. while this is rather unlikely to convince a teacher, if the students know that they are being scored by a system they might take their chances. publicly available content scoring systems in the automated student assessment prize (asap-sas) 1 have been shown to be vulnerable to such answers generated with cheating strategies like repeating the answer for multiple times and adding question-related or general academic words @xcite . what if the students don't need any strategies at all, but trick the systems with only some extreme irrelevant adversarial input? in this paper, we aim to investigate whether the need for generalization abilities (due to language errors and many ways to express the same fact) makes content scoring systems vulnerable to adversarial attacks. for this purpose, we generate adversarial answers based on different methods of increasing sophistication. we find that both shallow and deep learning scoring models are surprisingly vulnerable. even with simple techniques, up to 60% of generated wrong answers are accepted by the systems as being at least partially correct. this includes extreme examples where a string of random characters like "nswvtnvakgxpm" (also referenced in the title of this paper) was accepted as a fully correct answer. the adversarial is accepted as it contains some predictive, but non-robust feature @xcite like the character bigram 'kg'. as such system behavior is threatening the validity of the scoring system, we explore countermeasures to make models more robust. we find that removing vulnerable features like character n-grams and adversarial training can improve the robustness of shallow systems against adversarial answers. however, these countermeasures often come at the cost of a performance decrease on real student answers. this paper aims to bring the importance of system robustness against adversarial inputs in content scoring to the forefront. we also make our corpus of adversarial answers as well as their generation methods publicly available for future research. ## generating adversarial answers adversarial examples are instances with intentional perturbations that cause a machine learning model to make a false prediction @xcite . many prominent examples come from the image processing domain, e.g. a stop sign that was perturbed with little patches so that autonomous cars would classify it not as a stop sign (and possibly causing severe accidents). @xcite shows that googlenets classification of an image as 'panda' can be changed into 'gibbon' by adding a small vector. this kind of perturbation is usually imperceptible to humans, but decreases the performance of classification models drastically. following the similar principle in nlp applications, hotflip @xcite generates textual adversarial examples by swapping one character with another by gradient computation, and textbugger @xcite by inserting spaces or swapping random letters. these methods are quite likely to leave the semantics of the underlying text unchanged, which is much harder to achieve when texts are manipulated on the token level. a common strategy is to replace single words (usually nouns) with near-synonyms chosen by humans @xcite or as nearest neighbors in an embedding space @xcite . at the sentence level, scpns @xcite produce a paraphrase of a given sentence without changing the original meaning. in the educational domain, this is comparable to unusual, atypical or creative correct answers that run the risk of being classified as incorrect @xcite . however, none of the generation methods above is suitable to our research. if we use a real, correct answer as starting point for transformation into an adversarial one and leave the semantic meaning unchanged, we generate a correct answer. but we need answers which are definitely wrong in content as adversarial answers, in order to test the system's ability of rejecting cheating behaviour. we start with basic methods like uniform selection of characters and words from a generic corpus, but also use methods that are specific to the educational task like shuffling words in correct answers. the nonsense answer, generated by the most basic methods, are treated as a "worst case scenario". ## experimental setup to evaluate the robustness of automatic content scoring systems, we used the original training data in asap-sas to train several scoring models and tested each model on the generated adversarial examples as described in the previous section. ## results in this section, we apply the shallow and deep scoring systems to the adversarial examples. a perfect system should reject them with 100% arr, but figure 1 shows that this is not the case at all. on average, the shallow system can reject around 77% adversarial examples, while the deep system has an adversarial rejection rate of only 53%. on the same type of adversarial examples, the shallow system is more robust against adversarial answers than the deep one (the yellow diamond is always above the blue one). comparing the adversarial examples generated with character/word n-gram from the generic and the prompt-specific corpus, we find that both systems have a lower arr on adversarial answers generated on the basis of the asap-sas corpus. this supports our hypothesis, that adversarial answers with more prompt-specific words are more easily accepted. examples generated by shuffling have the lowest average arr on the shallow system, presumably because all the lexical material of a correct answer is still present in the corresponding adversarial example. if we look at the arr of adversarial examples generated with prompt-specific character n-gram combinations, increasing n is associated with decreasing performance for both systems. because examples generated with a larger n contain more possible real words, the models may treat these tokens as features indicative of correct answers. we are surprised to find that even random combinations of characters and words can be accepted as correct answers in some vulnerable prompts. the shallow system's model for prompt 2 is the most vulnerable model (shown as the yellow point at the bottom of the graph), while the deep system has the worst performance on prompt 10 (blue outlier). they both misclassify more than 90% random character answers as correct. to investigate the reason for the systems' vulnerabilities, we looked into the shallow system's worst performing model (on prompt 2), which scored more than 90% of adversarial answers generated with random characters as correct. an example adversarial response is nswvtnvakgxpm. analyzing the feature representations, we found that character bigrams "ns", "nv" and "kg" are the top 3 features found in the adversarial answers misclassified with the highest score. these frequent character bigrams are likely found in words related to correct answers in the training data such as "constant", or "investigation", while "kg" is an important word in itself. ## countermeasures we have seen that the content scoring systems used in the previous experiments are indeed susceptible to the adversarial examples generated with even simple methods. this section thus investigates countermeasures to make the systems more robust against such attacks. we consider three types of countermeasures: first, we explore the inclusion of features that we hypothesize to be particularly robust against adversarial asnwers and the exclusion of certain feature groups that we found to be particularly susceptible to adversarial answers. second, we enrich our training datasets in a data augmentation approach to include adversarial training examples. third, we use a filtering approach where we flag potential adversarial answers for human inspection in a separate module before the actual scoring. ## conclusion our results demonstrate that automatic scoring systems are extremely vulnerable to adversarial examples generated with simple methods. if the adversarial examples contain vocabulary from the prompt, they are even more damaging. the deep learning system tested in our study was more vulnerable than the shallow learning system. we found that adversarial training is the most promising countermeasure, but it has a small negative impact on task performance and the resulting system is still vulnerable. the results above place a greater demand on future research on automatic content scoring. instead of only improving scoring performance on real student answers, we should at the same time put more emphasis on the robustness against adversarial answers. otherwise, we risk developing methods that are better suited to the particular set of answers that happen to be in our evaluation dataset, but less robust against unexpected and adversarial input. with countermeasures like feature selection, adversarial training and filtering, a system can become more robust to the adversarial examples, but at the cost of a decreased classification performance on real student answers. limitations this study has limitations. on one hand, the adversarial examples were only tested on two content scoring systems. other systems with different feature sets or architectures might behave differently. however, we selected typical systems, to show typical behavior. thus, it is unlikely that the problem is going away completely. on the other hand, the generation methods for adversarial examples employed in this investigation are also limited. there might be other ways to trick the systems that we have not foreseen. future work in future work, we plan to further investigate the countermeasures for deep systems. as adversarial examples generated with random methods are easily recognized by humans, a semi-automatic assisted scoring scenario with a human in the loop might be explored to filter out obvious adversarial answers leaving only those examples easy for the machine but hard for humans for automatic detection.