Word-level Perturbation Considering Word Length and Compositional Subwords (2022)

Abstract: We present two simple modifications for word-level perturbation: Word Replacement considering Length (WR-L) and Compositional Word Replacement (CWR).In conventional word replacement, a word in an input is replaced with a word sampled from the entire vocabulary, regardless of the length and context of the target word.WR-L considers the length of a target word by sampling words from the Poisson distribution.CWR considers the compositional candidates by restricting the source of sampling to related words that appear in subword regularization. Experimental results showed that the combination of WR-L and CWR improved the performance of text classification and machine translation.

## introduction word-level perturbation is a well-known technique used nlp @xcite . for example, word replacement (wr) @xcite randomly replaces words in the input sequence with words sampled from a vocabulary. the conventional wr uses a uniform distribution for sampling. although a simple method, it is as effective as complex methods, such as adversarial perturbations @xcite . however, the conventional wr frequently replaces original words with unrelated words. if the probability of replacement (hyperparameter) is set to be large, a perturbed input sequence would be drastically different from the original one, which would significantly affect performance. thus, it is important to search for an appropriate hyperparameter. subword regularization (sr) @xcite @xcite ) is another effective method for word-level perturbation. we used different tokenizations sampled from a pretrained language model in each training epoch with the sr. as this method focuses only on tokenization, unrelated words are not used. however, sampling tokenization takes a longer time owing to its complex procedure for managing various tokenization candidates. in addition, the improvement achieved by sr is sometimes unimpressive in comparison with wr; however, it requires a considerable amount of time. in this study, we propose two approaches to compromise between wr and sr. our method restricts candidates in wr to related words in terms of (1) word length and (2) tokenization. the first approach weights the distribution for word sampling based on the length of the target word. the second approach hardly restricts the vocabulary for word sampling to compositional subwords of the original word inspired by sr. these restrictions prevent the replacement of words with unrelated words and thus result in a stable improvement in nlp tasks even if the hyperparameter is varied. in addition, the sampling speeds of our methods are faster than those of sr because they do not require an alternative tokenization sequence. we empirically demonstrate the advantages of the proposed method for text classification and machine translation tasks. ## related work this work discusses the technique of word-level perturbation in nlp. one of the popular perturbation ways is word replacement @xcite , which randomly choices input words and replaces them with other words in a vocabulary. word dropout @xcite and unknown token replacement @xcite are variations of word replacement, which replace the selected words with zero embeddings and unknown tokens, respectively. there are some techniques to prevent using unrelated words in word replacement. @xcite replaces randomly selected words with their synonyms. @xcite employs a language model to replace the chosen words. our work focuses on the tokenization units to restrict vocabulary to prevent using unrelated words. subword regularization is another means of word-level perturbation. @xcite employs a unigram language model to sample tokenization for machine translation. @xcite modifies byte pair encoding to perturb the input tokenization. @xcite @xcite introduces a technique to update the tokenizer during the training. ## proposed method 1 before describing our method, we provide a brief overview of the base method: wr. let @xmath0, ...x i , ...x i be a sequence of words whose length is i. the wr method randomly replaces x i with xi with probability a using the following equations: ## experiment we conducted experiments on text classification and machine translation. to confirm the effectiveness of our methods, we compared our method with regular training without word-level perturbation (vanilla) and the following four word-level perturbation techniques in addition to wr: subword regularization (sr) samples the tokenization in each training epoch with the pretrained unigram language model. we employed sentence-piece @xcite for sr. word dropout (wd) randomly replaces inputs with zero vectors @xcite . unknown token replacement (utr) randomly replaces words with unknown tokens @xcite , i.e., we use an unknown token as xi in eq.2. language model (lm) randomly replaces words with words sampled depending on an lm 4 . in addition to the proposed methods, wr-l and cwr, we denote the combination of these methods as cwr-l. ## conclusion we propose a fast and effective alternative for word-level perturbation. the experimental results showed that the proposed method, cwr-l, improved the performance of text classification and machine translation, particularly with the sampling strategy using poisson distribution. we also empirically showed that cwr-l is more robust to hyperparameters than other perturbation methods and is faster than sr.