UEGP : Unified Expert-Guided Pre-training for Knowledge Rekindle (2024)

Abstract: Pre-training and fine-tuning framework has become the standard training paradigm for NLP tasks and is also widely used in industrial-level applications. However, there are still a limitation with this paradigm: simply fine-tuning with task-specific objectives tends to converge to local minima, resulting in a sub-optimal performance. In this paper, we first propose a new paradigm: knowledge rekindle, which aims to re-incorporate the fine-tuned expert model into the training cycle and break through the performance upper bounds of experts without introducing additional annotated data. Then we further propose a unified expert-guided pre-training (UEGP) framework for knowledge rekindle. Specifically, we reuse fine-tuned expert models for various downstream tasks as knowledge sources and inject task-specific prior knowledge to pre-trained language models (PLMs) by means of knowledge distillation. In this process, we perform multi-task learning with knowledge distillation and masked language modeling (MLM) objectives. We also further explored whether mixture-of-expert guided pre-training (MoEGP) can further enhance the effect of knowledge rekindle. Experiments and analysis on eight datasets in GLUE benchmark and a industrial-level search re-ranking dataset show the effectiveness of our method.

## introduction in recent years, pre-trained language models (plms) have been widely used in various nlp tasks, such as sentiment classification, semantic matching, named entity recognition and etc., which generally adopt a two-stage training paradigm, i.e., pre-training and fine-tuning @xcite general plm task-specific expert model supervised fine-tuning unlabeled general corpora labeled data for specific tasks plm (a) traditional pre-training and fine-tuning paradigm general plm task-specific expert model supervised fine-tuning rekindle data plm (b) knowledge rekindle paradigm expert-guided plm infer knowledge transfer ## conclusion in this paper, we first propose knowledge rekindle paradigm as an improved paradigm of pre-training and fine-tuning, which aims to re-incorporate the fine-tuned expert model into the training cycle to further break through the performance upper bounds. we further propose a unified expert-guided pre-training method for knowledge rekindle, which adopts the combined objectives of knowledge distillation and masked language modeling. on the one hand, it enables plms to learn prior knowledge of downstream tasks, and on the other hand, it can avoid the model from converging to local minima in advance. in short, our method can exploit the performance upper bounds of plms without introducing additional data, and it is compatible with any plms and can be used sequentially. extensive experiments on the glue benchmark and industrial search re-ranking dataset demonstrate the effectiveness of our method.