Learning the Legibility of Visual Text Perturbations (2023)

Abstract: Many adversarial attacks in NLP perturb text in puts to produce visually similar strings (‘ergo’, ‘εrgo’) which are legible to humans but degrade model performance. Although preserving legibility is a necessary condition for text perturbation, little work has been done to systematically characterize it; instead, legibility is typically loosely enforced via intuitions around the nature and extent of perturbations. Particularly, it is unclear to what extent can inputs be perturbed while preserving legibility, or how to quantify the legibility of a perturbed string. In this work, we address this gap by learning models that predict the legibility of a perturbed string, and rank candidate perturbations based on their legibility. To do so, we collect and release LEGIT, a human-annotated dataset comprising the legibility of visually perturbed text. Using this dataset, we build both text- and vision-based models which achieve up to 0.91 F score in predicting whether an input is legible, and an accuracy of 0.86 in predicting which of two given perturbations is more legible. Additionally, we discover that legible perturbations from the LEGIT dataset are more effective at lowering the performance of NLP models than best-known attack strategies, suggesting that current models may be vulnerable to a broad range of perturbations beyond what is captured by existing visual attacks.

## introduction to manage the increasing demand for content moderation-e.g., detecting spam or toxic/hateful content on online platforms-organizations have turned to machine learning solutions. in response, users often resort to manipulating text to evade detection, removal, or search. for instance, hateful comments often comprise of visually similar characters to avoid automatic filtering @xcite . since people read text visually, the manipulated content can still be easily understood and harm its target audience. these attacks started with simple ascii substitutions like he11o (colloquially referred to as "leetspeak"), but have evolved into complex manipulations utilizing characters from different unicode scripts @xcite . figure 1 shows two such examples. unlike computer vision where there is an established notion of what constitutes an imperceptible perturbation (typically defined via the ℓ ∞ distance), most perturbations in text are perceptible. however, as long as the perceptible manipulations remain legible, the message could have its intended effect. the legibility of a text is determined by whether or not a literate person can decipher the altered words. the degree to which a piece of text can be perturbed, while maintaining legibility, depends on a multitude of factors such as its context, similarity to the original content, the positions of the perturbations, the background knowledge of the reader, etc. however, many adversarial attacks enforce legibility only loosely based on intuitions about the nature of the attacks, e.g., that changing 1-2 characters in a sentence does not impact its legibility @xcite . in this work, we instead propose to learn the legibility of visual perturbations, by developing textand vision-based models trained on legibility annotations from human subjects. the current focus of research on adversarial attacks is to find minimal perturbations required to break nlp models, and several recent findings suggest that models remain brittle to such perturbations @xcite @xcite . in contrast, our work attempts to uncover the space of all legible perturbations that we need to defend against. towards our goal of characterizing the limits of legibility of perturbed texts, we make the following contributions: first, we crowdsource human judgments about the legibility of different perturbations: specifically, we show annotators two perturbed versions of the same word and ask them which one, if any, they find more legible. our perturbation strategy considers substituting letters in the word with unicode characters drawn from a large subset of the basic multilingual plane covering over 100 scripts from around the world. 2 in total, we collect 30, 320 annotations, one each for 14, 643 and 3, 332 instances in the training and validation sets, respectively, and three each for the 4, 113 instances in the test set. using these preferences, we define a pairwise legibility ranking task as well as a binary legibility classification task. while the former allows making inferences about which candidate perturbation is most legible, the latter allows filtering out illegible perturbations altogether. for each task, we identify a hard subset of the collected data, which includes fine-grained comparisons expected to be more challenging for annotators and models alike. second, we use the labeled data to train models which predict the degree of legibility of a perturbed text. specifically, we fine-tune pretrained vision (trocr; @xcite ) and text-based (byt5; xue et al., 2022) models on the ranking and classification tasks. we find that trocr trained in a multi-task setup on both tasks achieves the best performance with 0.91 f1 score on the classification task and 0.86 accuracy on the ranking task. interestingly, we find that the purely text-based byt5 also achieves competitive performance on the classification task with 0.89 f1, suggesting that its pretrained byte representations encode aspects of visual similarity between unicode characters. further, we find that models have high f1 scores on the subset of data with high inter-annotator agreement: trocr achieves a 0.96 f1 score on test cases where all three annotators agree. we also note that legibility is a complex phenomenon-it doesn't correlate trivially with the distance of the perturbation from the original text or the number of letters substituted. 3 third, we consider a word-level perturbation recovery task, which involves inferring the original word from its perturbed version. we evaluate gpt-3 @xcite on this task, comparing its performance on legible perturbations from our perturbation strategy versus those generated by viper, a visual perturber method proposed by @xcite . we find that gpt-3 has a lower accuracy in recovering perturbations from our perturbation strategy, despite viper providing no guarantees on legibility. additionally, we apply our findings to the important task of toxicity classification. we perturb a subset of the dataset using our perturbation strategy and find that it degrades the sota detoxify (hanu and unitary team, 2020) classifier more than existing viper attacks. these findings demonstrate that existing attacks do not comprehensively cover the space of legible perturbations that can degrade model performance. ## related work adversarial attacks for nlp. a challenge in defining adversarial examples for text lies in characterizing the space of equivalent inputs to a training or test example which preserves the target label. while early work focused on adding distracting text to fool question answering systems @xcite , recent work utilizes more general strategies applicable to many tasks @xcite @xcite . many of these can be categorized as word-level synonym substitutions @xcite @xcite , or characterlevel legibility-preserving substitutions @xcite . most attacks in either category are perceptible in that readers of the text can identify that it has been transformed, except for one notable exception where invisible characters and near-identical characters are used to render strings indistinguishable from the original @xcite . attacks based on visual similarity of characters have also been previously considered by @xcite who propose three attack strategies: ices (based on rendered glyph similarity), dces (based on bag-of-words textual similarity of unicode codepoint descriptions), and eces (based on adding diacritics to base characters). for ices, they compute similarity by comparing raw pixel values of the renderings, which we improve upon here by utilizing a pretrained optical character recognition (ocr) model. this produces a 'smarter' set of visual neighbors: e.g., mirror images of letters, scaled versions of letters (like o vs • ) etc., which go beyond simple accents or modifiers. we also report in-depth comparisons between our perturbation strategy and the eces and dces approaches in section 5. legibility of perturbed inputs. among character-level perturbation attacks, legibility has only been loosely enforced based on intuitions about the nature and the degree of manipulations. this often results in conservative substitutions which only represent a lower bound on the space of all legible perturbations. for instance, pruthi et al., 2019 limit the attack to only 1-2 character changes (e.g., substitutions, deletions or additions) per input example; similarly, @xcite propose an attack strategy which specifically minimizes the number of character manipulations required in order to render the output legible. attacks based on visual similarity usually constrain their attack surface to inputs which are above a threshold similarity (in pixel or embedding space) to the original input @xcite @xcite . in this work, by contrast, we directly address the question of what constitutes legible perturbations, with the aim of learning a grounded definition of legibility rather than assuming one a priori. ## legibility tests we adopt a supervised learning approach for determining the legibility of perturbed texts. in this section, we describe the process used for collecting the legit dataset (which stands for legibility tests) and in the next section we describe the modeling techniques used for predicting the legibility score and ranking different candidate perturbations. our setting involves one-to-one character substitutions at the word level, i.e., given a word (and no other context), we consider perturbations where each letter in the word may be replaced by a unicode codepoint in 0x0000-0x2fff. moreover, the substitutions are mutually independent and do not depend on the context of the other letters. ## tasks and models in this section, we start by introducing two tasks for characterizing the legibility of perturbed texts, followed by a number of models for solving them. ## results classification task. for the classification task, we find that baselines that just use the metadata perform poorly. the majority class baseline obtains an f1 score of 0.677, and the logistic regression model using ϕ parameters yields an f1 score of 0.665, implying that legibility is not a simple function of the perturbation parameters k, n. the unsupervised vision-based models, imgdot and trocr embeddings, vastly improve upon the simple baselines, with the trocr embeddings obtaining an f1 score 0.868 and imgdot yielding an f1 score of 0.845. hence, these embeddings align reasonably well with human perceptions of legibility. the text-based byt5 models improve significantly over the baselines and unsupervised visionbased models. they are comparable to the performance of the single-task objective trocr-c, but worse than the trocr-mt. this suggests that the byt5 models might have encountered some visual perturbations during pretraining. single-task trocr-c model with the multi-task trocr-mt, we find that the presence of the additional ranking loss term during training improves model performance on the classification task from 0.886 to 0.905. on test examples where all 3 annotators agree, trocr performs even better, attaining an f1 score of 0.960, compared to a score of 0.850 on examples where only 2 annotators agree. as further evidence of the model's alignment with annotators, we find that the model confidence is directly correlated with annotator agreement (cf. figure 4 ) as measured by fleiss' κ @xcite . furthermore, consider figure 3 , which shows legibility scores obtained from trocr-mt for two words picked at random which are not part of the training set. qualitatively, we see that legibility scores from the trocr-mt model aligns with the human judgements of legibility for these words. ranking task. the trocr-mt model performs better relative to other models, resulting in a 6.8% absolute accuracy improvement. akin to the classification task, we find the trocr-mt model outperforms its single-task counterpart trocr-r. thus, training with a multi-task objective improves performance on both ranking and classification tasks when compared to single-objective models. differently from classification, we find that byt5 is significantly worse than the visionbased models on ranking, suggesting that language model pretraining is effective at separating legible from illegible perturbations, but not at encoding the degree of legibility of legible perturbations. jigsaw challenge. next, we check whether perturbations generated by our attack model ( § 3.1) and filtered to ensure legibility using trocr-mt are effective at degrading the performance of nlp models. we employ the jigsaw toxic comment classification dataset, which is a multilabel classification dataset consisting of wikipedia comments and human-annotated binary labels for 6 toxicity categories. in figure 4 (left), we compare legit and viper-dces strategies in a real-world scenario by perturbing the jigsaw dataset with each strategy and reporting how much these perturbations degrade the performance of detoxify-original (hanu and unitary team, 2020), a bert-based model which has state-of-theart performance on the jigsaw dataset. we show that legit produces greater degradation at lower n, and produces more legible perturbations even at higher n (due to trocr-mt filtering). in comparison, we find that dces perturbations become very hard to read at higher n, diluting the signifi-cance of the dces results at high n. appendix d provides a qualitative analysis of the legibility of dces perturbations compared to those generated using our legit method. viper-eces causes a negligible degradation on model performance, which is due to the fact that the bert tokenizer "corrects" almost all of the simple diacritic-based eces character substitutions. this means that the classification model receives mostly unperturbed input save for some isolated unks. for example, the perturbed input ť ĥâň ǩ ŷôǔ is tokenized back into thank you. taken together, these results demonstrate that legit exploits a more efficient legibility space, finding character substitutions which have a greater impact on model performance while preserving legibility. perturbing gpt-3. the strong performance of byt5 at separating legible from illegible inputs suggests that language models might be somewhat robust to such perturbations. to examine this, we experiment with gpt-3 (text-davinci-002 checkpoint) @xcite using a perturbation recovery task, wherein we prompt the model to decode perturbed words back to their original strings. we sample a subset of 1, 000 (w, w i ) pairs from legit which have a label of legible. these perturbations are fed to the gpt-3 model in batches of 10, along with an instructional prompt (see appendix e) and 4 examples; recovered words are received as a completion to the input prompt. in addition, we also perturb the same 1000 words using viper-dces and report the accuracy of gpt-3 at reconstructing them. we observe that gpt-3 often returns a word with a short edit distance to the original word, and hence to capture this in our evaluation, we apply the porter stemmer from nltk @xcite to both the original words and predicted reconstructions, and then measure how often their stemmed forms are the same. we repeat this experiment 3 times, randomly sampling the 4 examples in the prompt each time. figure 5 shows the gpt-3 accuracy at different fractions of corrupted characters (@xmath0.3, 0.7, 1.0}). as expected, the accuracy goes down as n increases, but we find that gpt-3 performs worse on legit perturbations. this demonstrates that while stateof-the-art language models are mildly robust to the narrower range of perturbations considered in existing visual attacks, they degrade significantly on inputs sampled from legit which are marked by humans as legible. this result underscores the importance of considering the entire space of legible perturbations when evaluating model robustness. ## conclusion we set out to characterize the limits of legibility of visual perturbations. to do so, we first collected and released a new dataset, legit, comprising legibility preferences of human subjects. using this dataset, we framed a binary legible-or-not classification task, and a ranking task to rank candidate perturbations. for these tasks, we explored several text-and vision-based models, and found that our models obtain a high f1 score of 0.91 for the classification task and an accuracy of 0.86 for the ranking task. perturbations generated using the same attack method as used for constructing legit lead to significant degradation on the jigsaw challenge task and are not recovered by gpt-3 accurately, despite being filtered for legibility. we believe this work opens avenues for research on legibility-driven certified robustness to visual attacks in nlp.