V ar MAE : Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding (2022)

Abstract: Pre-trained language models have been widely applied to standard benchmarks. Due to the flexibility of natural language, the available resources in a certain domain can be restricted to support obtaining precise representation. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token’s context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.

## introduction pre-trained language models (plms) have achieved promising performance in natural language understanding (nlu) tasks on standard benchmark datasets @xcite . most works @xcite leverage the transformer-based pre-train/fine-tune paradigm to learn contextual embedding from large unsupervised corpora. masked autoencoding, also named masked language model in bert @xcite , is a widely used pre-training objective that randomly masks tokens in a sequence to recover. the objective can lead to a deep bidirectional representation of all tokens in a bert-like architecture. however, these models, which are pre-trained on standard corpora (e.g., wikipedia), tend to underperform when migrated to a specific domain due to the distribution shift @xcite . recent works perform pre-training from scratch @xcite or continual pre-training @xcite on large domain-specific corpora. but in many specific domains (e.g., finance), effective and intact unsupervised data is difficult and costly to collect due to data accessibility, privacy, security, etc. the limited domain corpus may not support pre-training from scratch @xcite , and also greatly limit the effect of continual pre-training due to the distribution shift. besides, some scenarios (i.e., non-industry academics or professionals) have limited access to computing power for training on a massive corpus. therefore, how to obtain effective contextualized representations from the limited domain corpus remains a crucial challenge. relying on the distributional similarity hypothesis @xcite in linguistics, that similar words have similar contexts, masked autoencoders (maes) leverage co-occurrence between the context of words to learn word representations. however, when pre-training on the limited corpus, most word representations can only be learned from fewer co-occurrence contexts, leading to sparse word embedding in the semantic space. besides, in the reconstruction of masked tokens, it is difficult to perform an accurate point estimation @xcite based on the partially visible context for each word. that is, the possible context of each token should be diverse. the limited data only provides restricted context information, which causes maes to learn a relatively poor context representation in a specific domain. to address the above issue, we propose a novel variational masked autoencoder (varmae), a regularized version of maes, for a better domainadaptive language understanding. based on the vanilla mae, we design a context uncertainty learning (cul) module for learning a precise context representation when pre-training on a limited corpus. specifically, the cul encodes the token's point-estimate context in the semantic space into a smooth latent distribution. and then, the module reconstructs the context using feature regularization specified by prior distributions of latent variables. in this way, latent representations of similar contexts can be close to each other and vice versa @xcite . accordingly, we can obtain a smoother space and more structured latent patterns. we conduct continual pre-training on unsupervised corpora in two domains (science and finance) and then fine-tune on the corresponding downstream nlu tasks. the results consistently show that varmae outperforms representative language models including vanilla pre-trained @xcite and continual pre-training methods @xcite , when adapting to new domains with limited resources. moreover, compared with masked autoencoding in maes, the objective of varmae can produce a more diverse and well-formed context representation. ## varmae in this section, we develop a novel variational masked autoencoder (varmae) to improve vanilla mae for domain-adaptive language understanding. the overall architecture is shown in figure 1 . based on the vanilla mae, we design a context uncertainty learning (cul) module for learning a precise context representation when pre-training on a limited corpus. ## experiments we conduct experiments on science-and financedomain nlu tasks to evaluate our method. ## conclusion we propose a novel transformer-based language model named varmae for domain-adaptive language understanding with limited resources. a new cul module is designed to produce a diverse and well-formed context representation. experiments on science-and finance-domain tasks demonstrate that varmae can be efficiently adapted to new domains using a limited corpus. hope that varmae can guide future foundational work in this area.