Tailor: Generating and Perturbing Text with Semantic Controls (2022)

Abstract: Controlled text perturbation is useful for evaluating and improving model generalizability. However, current techniques rely on training a model for every target perturbation, which is expensive and hard to generalize. We present Tailor, a semantically-controlled text generation system. Tailor builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from semantic representations. We craft a set of operations to modify the control codes, which in turn steer generation towards targeted attributes. These operations can be further composed into higher-level ones, allowing for flexible perturbation strategies. We demonstrate the effectiveness of these perturbations in multiple applications. First, we use Tailor to automatically create high-quality contrast sets for four distinct natural language processing (NLP) tasks. These contrast sets contain fewer spurious artifacts and are complementary to manually annotated ones in their lexical diversity. Second, we show that Tailor perturbations can improve model generalization through data augmentation. Perturbing just ∼2% of training data leads to a 5.8-point gain on an NLI challenge set measuring reliance on syntactic heuristics.

## introduction semantic perturbation through controlled text generation modifies sentences to match certain target attributes, such as verb tense or sentiment (e.g., pos-@xmath0. it has been widely applied to a variety of tasks, e.g., changing text style @xcite , mitigating dataset biases @xcite , explaining model behaviors @xcite , and improving model generalization @xcite . existing efforts train task-specific generators, e.g., training figure 1 : a compositional perturbation using tailor. foot_0 given (a) an original sentence, we abstract each span into a structured header that contains its semantic roles and keywords. arguments to preserve are included in the context, along with blanks (<id_*>) denoting where new generated text may be inserted. we specify desired perturbations by modifying each control code (e.g., changing role locative)temporal in (b), verb tense past)present, and patient keyword specificity complete)partial). given these perturbed control codes in the input (c), tailor generates a new sentence (d) that reflects the desired perturbations. a sentiment style transferer requires instances annotated with positive and negative labels @xcite . as a result, they require costly annotated data and re-training for every task of interest. this work introduces tailor, a system that supports application-agnostic perturbations. at its core is a controlled generator ( §2) that flexibly generates outputs from target semantic attributes, which we represent through structured control codes in the inputs. as shown in figure 1 , these control codes build on the propbank semantic analysis @xcite of the original sentence: for each argument span, the semantic role and keyword control codes specify the desired semantic content for the span at varying levels of granu-larity. to encourage control code following, we train the tailor generator with unlikelihood training @xcite to penalize generations that are not aligned with designated control codes. the use of semantic role control codes allows tailor to perform fine-grained changes to individual arguments in a sentence (e.g., one can change only the patient in figure 1 ). instead of specifying a perturbation with a generic target property (e.g., positive)negative), we can specify the linguistic transformation used to achieve the property (e.g., changing sentiment through negation or antonym replacement). making such fine-grained perturbations allows for more careful evaluation and improvement of models' language understanding @xcite . to highlight the perturbations facilitated by tailor, we craft a list of primitive perturbation operations ( §3) on inputs to the generator; these can be easily composed to achieve more complex perturbations. in figure 1 , tailor transforms sentence a to d through a series of perturbations: syntactic rewriting (changing verb tense), then sentence expansion (extending "the athlete"), and finally data recombination (i.e., generating new text that contains "in" but follows the temporal control). compared to existing approaches that require training a separate model for every step or annotating a dataset that represents this transformation end-to-end, such compositions make tailor more cost-effective and generalizable. in fact, on nine fine-grained and compositional styleptb perturbations @xcite , tailor achieves performance compatible with task-specific baselines, and even outperforms them on five transfers ( §f). tailor's flexible and human-readable control codes allow for broad, easily extendable applicability. we demonstrate its utility in evaluating and improving nlp model robustness, showing that tailor can help replicate existing contrast sets on four diverse tasks. by abstracting manual perturbation types in prior work into perturbation strategies with tailor, we can apply the changes to larger datasets while saving manual annotation efforts. our analysis suggests that these contrast sets not only have high rates of validity, but also reduce spurious artifacts compared to the original evaluation datasets. in addition, tailor-produced contrast sets complement human annotated ones in terms of lexical diversity: only ∼10% of their unique tokens overlap with manually created contrast sets. we also explore tailor's utility in data augmentation. we find that augmenting training data with just ∼2% of tailor perturbations improves the robustness of natural language inference (nli) models to inference heuristics, increasing performance on the hans evaluation set @xcite by an average of 5.81 points and outperforming a previous syntactic augmentation method for nli. ## tailor's controllable generator here, we provide an overview of the tailor generator. we first outline three types of controls ( §2.1) that allow for specifying sentence meanings at varying granularity. next, we explain how to embed them within inputs ( §2.2) to the generator. we train the generator to follow control codes with unlikelihood training ( §2.3). ## creating perturbations with tailor with tailor, we can create diverse perturbations by modifying input controls. given an original sentence, we transform it to an input for tailor by extracting its semantic parses, foot_2 masking spans we wish to modify, and providing their control codes. then, we modify the control codes in the input to generate perturbed sentences with tailor, filtering out degenerate ones. primitive perturbation operations. we provide an easily-extendable set of perturbation macros, which capture three common types of perturbations in prior work, shown in while our control codes are mostly derived from semantic roles, these primitive operations broadly cover both syntactic and semantic changes. they can also be used in conjunction with external knowledge bases to achieve targeted edits. 5 , or be composed to achieve more complex perturbation strategies as shown in §5, §6, and appendix §f. ## intrinsic evaluation following previous work @xcite , we evaluate tailor generations on sentence likelihood, controllability, and closeness. ## contrast set creation manually creating contrast sets is expensive, e.g., @xcite reported spending 10-15 minutes per perturbation for ud parsing, whereas labeling existing data is more efficient @xcite . we show that tailor can reduce human labor by automatically generating contrast set instances such that annotators only have to label them. we create tailor-generated contrast sets for four stricting the perturbations to result in more plausible sets of controls. see §b for more details. ## data augmentation we explore whether tailor can be combined with noisy automated labeling for data augmentation. for the stanford natural language inference (snli) task @xcite , we show that data augmentation with tailor perturbations increases model robustness to inference heuristics. min et al. (2020) find that augmenting snli training data by swapping hypotheses' subject/objects (e.g., this collection contains 16 el grecos. 16 el grecos contain this collection) improves performance on hans, a challenge set for diagnosing fallible syntactic heuristics in nli models @xcite . following this, we use tailor to perturb hypotheses with the swap_core operation such that original @xmath1. we finetune roberta-base @xcite on different training datasets: original snli train data (unaugmented baseline), snli train augmented with min et al. (2020) (augmented baseline, referred to as syntactic perturb. in ## related work controllable text generation has been widely used to influence various properties of generated text for text summarization @xcite , data augmentation @xcite , style transfer (reid and zhong, 2021; madaan et al., 2020a), adversarial example generation @xcite , etc. most generators take simple controls like tense @xcite , topic @xcite , or sentiment polarity @xcite , which underspecify desired transformations. in contrast, tailor concretizes otherwise sparse controls (e.g., we can specify making a sentence more negative through negation.) recent works incorporating syntactic structures for paraphrasing @xcite @xcite @xcite or discrete semantic signatures for diverse generation @xcite are similar to tailor in their high-dimensional specification. also closely related are methods that reconstruct sentences from structured semantic representations. the most similar related work is infillmore @xcite , which uses semantic representations derived from framenet with constrained decoding to guide generation. while infillmore tunes the higher-level semantics of a sentence, tailor's semantic controls incorporate fine-grained information about the location and semantics of tex-tual phrases; in addition, we demonstrate two new applications for semantically-guided generation, contrast set generation and data augmentation. abstract meaning representation @xcite ) is an alternative semantic representation worth exploring for data perturbation, as it may further enable controls on entity recursions @xcite , though expressing such relationships is nontrivial. controlled generators have also been successfully used to perturb text for model training, evaluation, and explanation. they usually rely on application-specific labels @xcite @xcite or require pairs of original and perturbed sentences @xcite , which are expensive to generalize. also related are the creation of minimally edited datasets, either through manual rewriting @xcite , or creating perturbation templates @xcite @xcite ; tailor reduces the human efforts these studies require. ## discussion we propose tailor, a system that enables taskagnostic, complex and context-aware perturbations. tailor demonstrates that it is possible to drive fine-grained perturbations with semantic features directly derived from an instance. crucially, it shows that incorporating classical linguistic structures with modern large-scale neural architectures is feasible: with the help of modern pretrained large models, propbank-style shallow semantic representations can help steer generation towards desired meanings. factors that affect tailor's capability. though broadly applicable, tailor's controllability and effectiveness vary for different inputs. first, creating automatic perturbations with tailor requires external srl predictors, which can be noisy on rare semantic roles or low-resource languages. 14 empirically, this did not seem to be a bottleneck, as exposing biases in downstream tasks does not usually require rarity at the semantic role level (e.g., testing syntactic heuristics in nli requires swapping only agents and patients). however, perturbing more challenging linguistic phenomena may re-quire careful srl predictor augmentation or even manual semantic role annotation. we also notice tailor can sometimes produce degenerate outputs. we hypothesize that this is a byproduct of unlikelihood trainingi.e., the generator learns to reduce the likelihood of negative sequences by generating tokens that are very unlikely to appear in natural text. generation hyperparameters (e.g., number of beams) can reduce the number of degenerate outputs. while we perform unlikelihood training at the sequence level, future work can investigate the effect of penalizing generation at the level of tokens or spans, which may provide finer-grained signals for which spans should be considered unlikely, as well as more strategically balancing positive and negative samples. extending tailor. we believe the tailor generator is well-suited for controlled generation tasks beyond the perturbation-based tasks we explore. given key entities or arguments as keywords and fully masked contexts, we envision tailor can help generate arguments @xcite , compositionally augment data @xcite , or generate captions @xcite . in particular, as shown in §5, tailor's human-readable controls can support humans on data curation, which suggests that designing nlp models for augmenting human capabilities is a promising direction. the design of controls is also worthy of in-depth exploration. as mentioned in §7, amr might be an alternative for semantic representation, if our primary goal is to express non-sequential relations. on the other hand, dependency parsing labels are useful for syntactic changes; future work may try to balance syntactic and semantic controls. having noted these opportunities, we believe tailor is already a powerful tool for perturbation, particularly for tasks where compositional changes are required. tailor is opensource, and available at https://github.com/allenai/tailor .