AMAL : Meta Knowledge-Driven Few-Shot Adapter Learning (2022)

Abstract: NLP has advanced greatly together with the proliferation of Transformer-based pre-trained language models. To adapt to a downstream task, the pre-trained language models need to be fine-tuned with a sufficient supply of annotated examples. In recent years, Adapter-based fine-tuning methods have expanded the applicability of pre-trained language models by substantially lowering the required amount of annotated examples. However, existing Adapter-based methods still fail to yield meaningful results in the few-shot regime where only a few annotated examples are provided. In this study, we present a meta-learning-driven low-rank adapter pooling method, called AMAL, for leveraging pre-trained language models even with just a few data points. We evaluate our method on five text classification benchmark datasets. The results show that AMAL significantly outperforms previous few-shot learning methods and achieves a new state-of-the-art.

## introduction since transformer-based @xcite pre-trained language models (plms) on massive corpora made a big impact on nlp, fine-tuning plms @xcite @xcite has led to large improvements in a variety of downstream nlp tasks. yet, it is still challenging to fine-tune plms @xcite in the few-shot regime. recently, adapters @xcite @xcite have provided a method of fine-tuning plms more efficiently, by tuning some extra weights (the adapters) while freezing the rest. nevertheless, existing adapters still fail to yield significant results in the few-shot regime. refer to the appendix in this paper, we present a cost-effective method for language model fine-tuning that is applicable, without customization, to a variety of language models and adapter types. we focus on small to mid-sized language models such as bert @xcite , roberta @xcite , albert @xcite , bart @xcite , or deberta @xcite , because they are widely deployed in production systems due to their economy and low carbon footprint. in this paper, we propose a meta-knowledgedriven few-shot adapter learning method, called amal (adapter-by-meta-learning), based on a novel meta-learning framework, through which meta-level layer-wise adaptation kernels are derived in an end-to-end manner. our design takes inspiration from @xcite , which proves that the over-parameterized pre-trained language models actually have low intrinsic dimension. we hypothesize that language model finetuning can be accomplished on a low intrinsic rank while keeping the pre-trained weights frozen, leading to our proposed low-rank adapter pooling approach. amal includes two key ideas: (1) construction of language model adapters' intrinsic kernels from tasks and (2) inference of the optimal task-specific language model adapter for a given task, by referring to a meta-level latent embedding space over all tasks. amal can be seen as similar with lora in terms of using the low-rank decomposition technique. however as a meta learning-based approach, amal can be applied to a broad range of language models and all existing adapter-based methods, including lora. ## proposed method: amal in this section, we present the implementation of amal. the design implies the hypothesis that the language model adaptation can be performed on a low intrinsic rank. here, we describe amal by employing the original adapter @xcite ) method. importantly, amal is orthogonal to existing adapter methods and can be combined with any of them. amal offers a taskspecific adapter for an incoming task. amal alternates between two update processes during meta-training: (1) low-rank adapter pooling and (2) meta-optimization. ## conclusion we hypothesized that language model adaptation can be performed on a low intrinsic rank, especially when only a few examples are offered. we designed a novel meta-learning-based low-rank adaptation method for leveraging small to mid-sized pretrained language models, allowing a new task to be cost-effectively learned in the few-shot regime. we demonstrated that the combination of low-rank matrix decomposition and meta learning is so effective, that we can reap the benefits of small to mid-sized pre-trained language models in practical scenarios with scarce annotated data.