Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning (2020)

Abstract: Pretrained Language Models (PLMs) have improved the performance of natural language understanding in recent years. Such models are pretrained on large corpora, which encode the general prior knowledge of natural languages but are agnostic to information characteristic of downstream tasks. This often results in overfitting when fine-tuned with low resource datasets where task-specific information is limited. In this paper, we integrate label information as a task-specific prior into the self-attention component of pretrained BERT models. Experiments on several benchmarks and real-word datasets suggest that the proposed approach can largely improve the performance of pretrained models when fine-tuning with small datasets.

## introduction recently, pretrained language models (plms) @xcite have yield significant progress on various natural language processing (nlp) tasks, e.g., neural language understanding, text generation, etc. existing plms are usually pretrained in a task-agnostic manner, in which the model is expected to capture the general knowledge of natural language from a large corpus, independent of downstream-specific information. this is not a problem when data is abundant in the downstream dataset, in which case, the model can effectively extract task-specific information during fine-tuning. however, in real scenarios, data may be difficult to collect and labeling is usually expensive. we show that plms pretrained with general knowledge can overfit without enough guidance from the task-specific information, resulting in degraded performance during testing. in this paper, we propose to utilize the label embeddings as a task-specific prior, complementary to the general prior already encoded during pretraining. we learn and integrate these label embeddings into bert models @xcite to regularize its self-attention modules, so the task-irrelevant tokens or patterns can be readily filtered out, while the task-specific information can be enhanced during fine-tuning. such a modification is compatible with any plm built upon self-attention and will not degrade the original pretrained structure. in order to validate the performance of our approach in a real-world setting, we collected two text classification datasets from the online patient portal of a large academic health system, each with a few thousand sequences. these are the first datasets for automatic patient message triage, which constitute an important problem in the field of clinical data analysis. experimental results show that our approach significantly improves the performance of fine-tuning on low-resource datasets, e.g., those consisting of only several thousand data samples. ## experiments we focus on fine-tuning with small datasets. we integrate label embeddings into the pretrained (bio)bert models, and fine-tune on various classification benchmarks as well as two real-world clinical datasets that we collected from the online patient portal of a large academic health system. ## conclusion we propose to integrate task specific information into plms that are pretrained with task-agnostic objectives. to do this, we leverage label embeddings to regularize the self-attention in plms. results on public benchmarks and real-world datasets suggest that our method can effectively improve the results for low resource fine-tuning. label count typical example non-urgent 631 that would be awesome... thank you. medium 955 dr. [name]. all seems well now. i am at home resting. my wife and i have a trip planned to maryland this week beginning on wednesday. we can fly, drive or stay home if i should not travel. are there any reasons that i should not fly. urgent 170 i have continued having chest pain shortness of breath since waking. please tell me what to do. i have tried in hailers am going to try nebulizers. i just feel extremely tight in my chest.