Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning (2023)

Abstract: Large language models (LLMs) have showcased their capability with few-shot inference known as in-context learning. However, in-domain demonstrations are not always readily available in real scenarios, leading to cross-domain in-context learning. Besides, LLMs are still facing challenges in long-tail knowledge in unseen and unfamiliar domains. The above limitations demonstrate the necessity of Unsupervised Domain Adaptation (UDA). In this paper, we study the UDA problem under an in-context learning setting to adapt language models from the source domain to the target domain without any target labels. The core idea is to retrieve a subset of cross-domain elements that are the most similar to the query, and elicit language model to adapt in an in-context manner by learning both target domain distribution and the discriminative task signal simultaneously with the augmented cross-domain in-context examples. We devise different prompting and training strategies, accounting for different LM architectures to learn the target distribution via language modeling. With extensive experiments on Sentiment Analysis (SA) and Named Entity Recognition (NER) tasks, we thoroughly study the effectiveness of ICL for domain transfer and demonstrate significant improvements over baseline models.

## introduction large language models (llms) have demonstrated remarkable success in various tasks via incontext learning (icl) with task instructions and few-shot demonstrations (input-label pairs) @xcite @xcite , eliminating the need for fine-tuning from task-specific labels. nevertheless, in-domain demonstrations are usually absent in real scenarios since the target domain labels are unavailable. sourcing labeled examples from other domains may suffer from huge syntactic and semantic domain shifts. moreover, ## problem definition consider a scenario where we have access to two distinct domains: a source domain and a target domain. the source domain dataset, denoted as d s , consists of n labeled data sampled i.i.d. from the source distribution, d @xmath0, y i } 1,...,n , where x s i represents sequences of tokens, y i represents the corresponding label. on the other hand, the unlabeled target domain dataset, denoted as d @xmath1,...,m , comprises m unlabeled data points, which are also sampled i.i.d. from the target domain. the primary objective of unsupervised domain adaptation (uda) is to adapt the knowledge learned from the source domain in such a way that allows it to generalize on the target domain effectively. this adaptation process involves leveraging the unlabeled data from the target domain to learn the target distribution and mitigate the domain shift. this paper focuses on the uda problem over two application scenarios: named entity recognition (ner) foot_0 and sentiment analysis (sa). we describe our method and pivot discussions around these two tasks in the following sections. ## method transformer-based encoder vx v[sep] v[mask] … e[sep] e[mask] crf source domain input: target domain unlabeled corpus: …… …… ………. ## conclusion in this work, we propose domain-adaptive incontext learning to acquire knowledge of both the target domain distribution and the discriminative task signal simultaneously. we develop different prompting and fine-tuning strategies that take into account various lm architectures and different language modeling mechanisms. overall, our framework demonstrates significant performance gains over an extensive spectrum of cross-domain experiments, and we perceive that fine-tuning is still effective and promising in the era of large language models when it involves domain shift.