H idden C ut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability (2021)

Abstract: Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples. We have publicly released our code at https://github.com/GT-SALT/HiddenCut .

## introduction fine-tuning large-scale pre-trained language models (plms) has become a dominant paradigm in the natural language processing community, achieving state-of-the-art performances in a wide range of natural language processing tasks @xcite @xcite @xcite @xcite @xcite . despite the great success, due to the huge gap between the number of model parameters and that of task-specific data available, the majority of the information within the multi-layer self-attention networks is typically redundant and ineffectively utilized for downstream tasks @xcite @xcite . as a result, after task-specific fine-tuning, models are very likely to overfit and make predictions based on spurious patterns @xcite , making them less generalizable to outof-domain distributions @xcite @xcite . in order to improve the generalization abilities of over-parameterized models with limited amount of task-specific data, various regularization approaches have been proposed, such as adversarial training that injects label-preserving perturbations in the input space @xcite @xcite , generating augmented data via carefully-designed rules @xcite @xcite , and annotating counterfactual examples @xcite . despite substantial improvements, these methods often require significant computational and memory overhead @xcite @xcite or human annotations @xcite . in this work, to alleviate the above issues, we rethink the simple and commonly-used regularization technique-dropout @xcite in pre-trained transformer models @xcite . with multiple self-attention heads in transformers, dropout converts some hidden units to zeros in a random and independent manner. although plms have already been equipped with the dropout regularization, they still suffer from inferior performances when it comes to out-of-distribution cases @xcite . the underlying reasons are two-fold: (1) the linguistic relations among words in a sentence is ignored while dropping the hidden units randomly. in reality, these masked features could be easily inferred from surrounding unmasked hidden units with the self-attention networks. therefore, redundant information still exists and gets passed to the upper layers. (2) the standard dropout assumes that every hidden unit is equally important with the random sampling procedure, failing to characterize the different roles these features play in distinct tasks. as a result, the learned representations are not generalized enough while applied to other data and tasks. to drop the information more effectively, @xcite recently introduce cutoff to remove tokens/features/spans in the input space. even though models will not see the removed information during training, examples with large noise may be generated when key clues for predictions are completely removed from the input. to overcome these limitations, we propose a simple yet effective data augmentation method, hid-dencut, to regularize plms during the fine-tuning stage. specifically, the approach is based on the linguistic intuition that hidden representations of adjacent words are more likely to contain similar and redundant information. hiddencut drops hidden units more structurally by masking the whole hidden information of contiguous spans of tokens after every encoding layer. this would encourage models to fully utilize all the task-related information, instead of learning spurious patterns during training. to make the dropping process more efficient, we dynamically and strategically select the informative spans to drop by introducing an attentionbased mechanism. by performing hiddencut in the hidden space, the impact of dropped information is only mitigated rather than completely removed, avoiding injecting too much noise to the input. we further apply a jensen-shannon divergence consistency regularization between the original and these augmented examples to model the consistent relations between them. to demonstrate the effectiveness of our methods, we conduct experiments to compare our hiddencut with previous state-of-the-art data augmentation method on 8 natural language understanding tasks from the glue @xcite benchmark for in-distribution evaluations, and 5 challenging datasets that cover single-sentence tasks, similarity and paraphrase tasks and inference tasks for out-ofdistribution evaluations. we further perform ablation studies to investigate the impact of different selecting strategies on hiddencut's effectiveness. results show that our method consistently outperforms baselines, especially on out-of-distribution and challenging counterexamples. to sum up, our contributions are: ## hiddencut approach to regularize transformer models in a more structural and efficient manner, in this section, we introduce a simple yet effective data augmentation technique, hiddencut, that reforms dropout to cutting contiguous spans of hidden representations after each transformer layer (section 3.1). intuitively, the proposed approach encourages the models to fully utilize all the hidden information within the self-attention networks. furthermore, we propose an attention-based mechanism to strategically and judiciously determine the specific spans to cut (section 3.2). the schematic diagram of hiddencut, applied to the transformer architecture (and its comparison to dropout) are shown in figure 1 . ## conclusion in this work, we introduced a simple yet effective data augmentation technique, hiddencut, to improve model robustness on a wide range of natural language understanding tasks by drop-ping contiguous spans of hidden representations in the hidden space directed by strategic attentionbased sampling strategies. through hiddencut, transformer models are encouraged to make use of all the task-related information during training rather than only relying on certain spurious clues. through extensive experiments on indistribution datasets (glue benchmarks) and outof-distribution datasets (challenging counterexamples), hiddencut consistently and significantly outperformed state-of-the-art baselines, and demonstrated superior generalization performances.