Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning (2023)

Abstract: As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the high cost of fine-tuning. While large PLMs and various PETL methods have achieved impressive results on various benchmarks, it is uncertain whether they can effectively handle inputs that have been distributionally shifted. In this study, we systematically explore how the ability to detect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer methods are altered. Specifically, we evaluated various PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, with various language models with different scales.

## introduction pre-trained language models (plm), which are pretrained on large-scale corpora using transformerbased architectures @xcite , have achieved groundbreaking success on sundry benchmarks @xcite @xcite , establishing themselves as the standard neural model in countless applications. moreover, language models pre-trained with larger parameters on a rich volume of corpora tend to exhibit more intriguing potentials, such as the ability to capture world knowledge @xcite , generate codes @xcite , and even solve mathematical problems @xcite , on top of understanding linguistic knowledge (e.g., semantic or syntactic). to explore the apex of pretrained language models (plms), the size of plms is growing exponentially and has reached billions to a trillion @xcite @xcite . under these circumstances, the conventional method for transferring plms to a target task (i.e., fine-tuning) is now infeasible as it entails prohibitive costs to train and store the entire parameters of large plms for every desired task. to mitigate this issue, several recent parameterefficient transfer learning (petl) methods have been proposed to improve task scalability. for instance, adapter-based @xcite approaches insert small neural modules into each layer of the plm and update those lightweight modules in the training phase. inspired by the recent success of textual prompts @xcite , prompt-based methods @xcite @xcite concatenate extra tunable tokens to the front of the input or hidden layers and update prepended soft prompts in the training phase. despite these breakthroughs in nlp, even very recent anomaly detection studies @xcite are still limited to relatively small bi-directional plms (e.g., bert, roberta). thus, how large-scale plms or auto-regressive plms cope with outliers is uncharted territory, naturally begging the following questions: • q1: does increasing model size improve ood detection performance without model parameters? to resolve these questions, this paper investigates the capability of large plms as outlier detectors from various perspectives. specifically, we compare the robustness to outliers with various transfer learning techniques on several ood bench-marks: full fine-tuning, lora @xcite , adapter @xcite , and prefix-tuning @xcite on various auto-regressive plms with different sizes, i.e., gpt2-s, m, l, xl @xcite , gpt-neo @xcite and gpt-j @xcite . from in-depth investigations, we share several intriguing observations: (1) as the size of the plm increases, the performance improves without any update of model parameters. however, it is still challenging to use it without supervision since their performances still lag far behind compared to the finetuned small plm (i.e., bert-base). (2) petls outperform fine-tuning with sufficiently large plms in both ind and ood metrics. (3) lastly, leveraging the information of the last hidden representation, which is the most prevailing method for bi-directional plm in recent ood detection, does not transfer well in auto-regressive plm, requiring a novel representation extracting technique. we believe that these findings will help future anomaly detection studies. ## analysis in this section, we share several intriguing findings and insights from various settings. ## conclusion and future work in this study, we showed that the scale of the language model is an important factor in ood robustness. moreover, we also showed that various methodologies outperform fine-tuning when applied to sufficiently large plm. our follow-up work seeks to create a methodology that allows large plms to be more robust to ood input. the performance improvement that can be achieved by the size of plm and ood technique is orthogonal. in line with the growing size of plm, the ood technique needs to be developed in a more parameterefficient way. as such, developing a proper ood technique compatible with the parameter-efficient transfer methods is our proper goal.