Snapshot-Guided Domain Adaptation for ELECTRA (2022)

Abstract: Discriminative pre-trained language models, such as ELECTRA, have achieved promising performances in a variety of general tasks. However, these generic pre-trained models struggle to capture domain-specific knowledge of domain-related tasks. In this work, we propose a novel domain-adaptation method for ELECTRA, which can dynamically select domain-specific tokens and guide the discriminator to emphasize them, without introducing new training parameters. We show that by re-weighting the losses of domain-specific tokens, ELECTRA can be effectively adapted to different domains. The experimental results in both computer science and biomedical domains show that the proposed method can achieve state-of-the-art results on the domain-related tasks.

## introduction pre-trained language models @xcite have demonstrated significant capabilities in various nlp tasks. while most language models follow the bert-style @xcite to predict original tokens of the masked positions, electra @xcite trains a discriminator to predict whether each token in a corrupted input is replaced by a generator. bert mainly learns from the masked subset of input tokens, but electra could predict all input tokens, significantly improving sample efficiency and leading to strong results on general tasks @xcite @xcite @xcite @xcite . domain adaptation of bert-style models has been proved to consistently improve on the domain-related tasks @xcite @xcite . however, the adaptation of electra is still under-explored. this leads us to investigate domain adaptation of ## conclusion in this paper, we design a snapshot-guided domain adaptation method for electra to capture the token-level domain knowledge by comparing generators of different training steps. our method can dynamically select and emphasize the domainspecific tokens, which can benefit domain adaptation. experimental results show that our method achieves state-of-the-art results without introducing additional training parameters.