Rethinking Denoised Auto-Encoding in Language Pre-Training (2021)

Abstract: Pre-trained self-supervised models such as BERT have achieved striking success in learning sequence representations, especially for natural language processing. These models typically corrupt the given sequences with certain types of noise, such as masking, shuffling, or substitution, and then try to recover the original input. However, such pre-training approaches are prone to learning representations that are covariant with the noise, leading to the discrepancy between the pre-training and fine-tuning stage. To remedy this, we present ContrAstive Pre-Training (CAPT) to learn noise invariant sequence representations. The proposed CAPT encourages the consistency between representations of the original sequence and its corrupted version via unsupervised instance-wise training signals. In this way, it not only alleviates the pretrain-finetune discrepancy induced by the noise of pre-training, but also aids the pre-trained model in better capturing global semantics of the input via more effective sentence-level supervision. Different from most prior work that focuses on a particular modality, comprehensive empirical evidence on 11 natural language understanding and cross-modal tasks illustrates that CAPT is applicable for both language and vision-language tasks, and obtains surprisingly consistent improvement, including 0.6% absolute gain on GLUE benchmarks and 0.8% absolute increment on NLVR2.

## introduction recently, pre-trained self-supervised models such as bert @xcite have attracted an increasing amount of attention in natural language processing and vision-language processing. benefiting from common knowledge contained in massive unlabeled data @xcite , the pretraining-finetuning framework has become most endeavors on pre-trained representation models rely on elaborately designed selfsupervised tasks, which typically corrupt the given sequence with certain types of noise (e.g., masking in bert in to remedy this, we present contrastive pre-training (capt) to learn noise invariant (or denoised) sequence representations. the core idea of capt is to enhance the consistency between semantic representations of the original sequence and that of corresponding corrupted version (e.g. the masked sequence) via unsupervised instancewise training signals. as shown in figure 1 , our approach strives to pull the representation of the corrupted sequence towards that of the original instance in the semantic space, while pushing it away from representations of other instances. such training objectives are formulated as a multi-class classification task, which aims at classifying the original sequence to the class of its corrupted version and vice versa, while classifying different instances into different classes. moreover, in order to enable the model to learn from more "difficult" and "diverse" instances, two effective methods are proposed to further enhance the capability of the model to extract noise-concentrated and instancediffused features. with such training objective, the pre-trained model is encouraged to learn noise invariant representations, thereby alleviating the pretrain-finetune discrepancy to some extent. as an additional benefit, our approach also assists the pre-trained model to more effectively capture the global semantics of the input. most prior work only focuses on token-level pre-training tasks (e.g. masked language modeling), which lacks the modeling of global semantics of the input. some other efforts alleviate this problem by introducing sentence-level pre-training tasks (e.g. next sentence prediction) that rely on the relative position of segments in the document. however, the semantic connection between these segments tends to be excessively loose, which may result in confusing gradient signals @xcite . by contrast, our capt offers incentives for representations of inputs sharing the same semantics (the original instance and its corrupted version) to be similar, while the representations of inputs expressing dif-ferent semantics (different instances) are penalized to be distinguished from each other. such more reasonable sentence-level supervision enables our approach to look beyond the local structures of input sequences and become more aware of the global semantics. we perform the evaluation on a comprehensive suite of benchmark, covering 8 natural language understanding and 3 cross-modal tasks. extensive empirical evidence demonstrates that our approach can achieve consistent improvements over the baselines in both language and vision-language domains. to be more specific, our capt raises the performance of roberta @xcite from 88.9% to 89.5% on the glue dev set, and also surpasses lxmert @xcite ) by 0.5%, 0.6% and 0.8% on vqa, gqa and nlvr 2 , respectively. ## experiments on language tasks this section presents experiments on language tasks, including specific implementation and detailed results. ## conclusion this work presents contrastive pre-training for learning denoised sequence representations in a self-supervised manner. by enhancing the consistency between representations of the original sequence and the corresponding corrupted version, the pre-trained model is encouraged to learn noise invariant sequence representations. on this account, the proposed approach not only alleviates the pretrain-finetune discrepancy induced by the noise of pre-training, but also better captures the global semantics of the input via more effective sentence-level supervision. extensive experiments demonstrate the effectiveness and versatility of our approach, which can achieve consistent improvements over baselines in both language and visionlanguage domains.