Compositional Zero-Shot Domain Transfer with Text-to-Text Models (2023)

Abstract: Label scarcity is a bottleneck for improving task performance in specialized domains. We propose a novel compositional transfer learning framework (DoT51) for zero-shot domain transfer. Without access to in-domain labels, DoT5 jointly learns domain knowledge (from masked language modelling of unlabelled in-domain free text) and task knowledge (from task training on more readily available general-domain data) in a multi-task manner. To improve the transferability of task training, we design a strategy named NLGU: We simultaneously train natural language generation (NLG) for in-domain label-to-data generation, which enables data augmentation for self-finetuning and natural language understanding (NLU) for label prediction. We evaluate DoT5 on the biomedical domain and the resource-lean subdomain of radiology, focusing on natural language inference, text summarization, and embedding learning. DoT5 demonstrates the effectiveness of compositional transfer learning through multi-task learning. In particular, DoT5 outperforms the current state-of-the-art in zero-shot transfer by over 7 absolute points in accuracy on RadNLI. We validate DoT5 with ablations and a case study demonstrating its ability to solve challenging NLI examples requiring in-domain expertise.

## introduction while pretrained language models demonstrate massive improvements on a wide range of natural language processing (nlp) tasks, it remains challenging to apply them to specialized domains @xcite . to acquire domainspecific task knowledge, a conventional approach is to perform domain-specific pretraining-usually masked language modelling (mlm) on in-domain raw text-followed by finetuning with in-domain task-annotated data @xcite @xcite . however, this approach requires in-domain task labels that can be expensive to acquire. another approach is to train a model with the usually abundant general-domain task labels and directly transfer to the new domain @xcite , but the transfer performance is often limited by the domain gap. past studies on zero-shot domain transfer or unsupervised domain adaptation have explored methods to transfer task knowledge from a source domain to an unseen target domain @xcite , but they usually require external modules to perform feature or domain alignment and are not always easily applicable to pretrained language models. in particular, there is little understanding of how we can leverage and combine domain-specific knowledge and general-domain task knowledge in the context of the recent success of text-to-text architectures in transfer learning. to close this gap, we propose dot5, a novel compositional zero-shot domain-transfer framework based on the state-of-the-art (sota) transfer learning model transfer learning model textto-text transfer transformer (t5) @xcite . throughout, the 'zero-shot' setup refers to zero-shot domain transfer with no access to labelled in-domain data. 2 by ''compositional'' we mean that dot5 is able to combine seen task figure 1 : by combining task knowledge from general domain data and domain knowledge from in-domain unlabelled text, our text-to-text model dot5 learns to solve in-domain tasks. labels and domain text to acquire an unseen combination of task domain knowledge. as shown in figure 1 , dot5 combines domain knowledge and task knowledge by making the best use of in-domain free text and general-domain task labels, which are typically accessible and abundant. for example, in the context of natural language inference (nli), dot5 can learn domain-specific semantics (e.g., ''bony abnormalities'' is a synonym of ''osseous abnormalities'') from in-domain free text and transferable task knowledge from general-domain task labels (e.g., negation indicates contradiction) to infer domain-specific task knowledge (e.g., ''there are no bony abnormalities'' contradicts ''there are osseous abnormalities''). we apply dot5 to nli, summarization, and text embedding learning, which are fundamental applications across many domains, and we explore zero-shot domain transfer to the high-value and highly specialized domain of biomedicine and its extremely low-resource subdomain of radiology. due to their specialization, obtaining labelled data in these domains is expensive and time-consuming. for example, the radiologyspecific nli dataset (radnli) @xcite contains only 960 manually labelled examples as development and test data and no training data is available. the key to dot5's compositional transfer is continual multi-task pretraining to simultaneously acquire domain and task knowledge: we jointly train t5 with mlm on in-domain unlabelled data and general-domain tasks @xcite . to better acquire the transferable task knowledge from the general-domain task labels, we propose a multi-task setup we call nlgu. as depicted in figure 2 , nlgu gives each task two formulations: natural language generation (nlg) @xcite , and natural language understanding (nlu) (data-to-label prediction). nlu enables label prediction when tested in an unseen domain and forces model sensitivity to the conditioned label, assisting nlg. meanwhile, nlg enables downstream tasks such as summarization or data augmentation. this enables dot5 to generate its own nli in-domain task data for further finetuning (a process we call selffinetuning), or to generate positive and negative examples for improving text embeddings by contrastive learning @xcite . our experiments show the effectiveness of dot5 in zero-shot domain transfer, and our proposed multi-task compositional approach achieves large gains compared with sequential training with t5 across all tasks. in particular, we achieve sota zero-shot domain transfer performance on radnli @xcite , outperforming baselines including large language models (llms), sequential training approaches, and task-specific baselines by large margins. we also identify several key insights through extensive analysis: 1) all three key components (in-domain mlm, nlgu, self-finetuning) in dot5 are important for transfer success while multi-task learning with in-domain mlm is the key for combining domain and task knowledge. 2) scaling up model size significantly improves transfer performance. ## method to achieve compositional transfer, dot5 acquires domain knowledge and task knowledge via continual pretraining (see figure 2 ). specifically, we optimize a joint loss function composed of an in-domain masked language model loss (''domain-mlm'') and a general-domain taskspecific loss: we set λ = 0.5 but explore tuning it in §4.1. we use t5, an encoder-decoder generative language modelling framework @xcite , to learn a conditional sequence generator p (output|input). t5 is chosen for two reasons: 1) it is a strong transfer learning model, and 2) it can unify classification and generation, which has potential to further boost transfer performance (see nlgu discussion in §3.2). we use the same pretraining objective (cross-entropy with teacherforcing) as in t5. we detail the two loss components for continual pretraining in §3.1 and §3.2. once the model has been continually pretrained, it can be used to perform zero-shot domain transfer on a task. taskspecific designs for inference are given in §3.3. ## experiment we introduce our experimental setup in §4.1, briefly discuss baseline approaches in §4.2, and present results in §4.3. ## further analysis in this section, we demonstrate the importance of individual components of dot5 ( §5.1) and explore the role of model size ( §5.2). finally, we provide fine-grained analysis on radnli to verify whether dot5 has indeed acquired domainspecific task knowledge from compositional transfer ( §5.3). ## conclusion and discussion we propose dot5, a compositional transfer learning framework to solve domain-specific nlp tasks without requiring in-domain task labels. we show the effectiveness of dot5 on zero-shot transfer to multiple tasks in the biomedicine and radiology domains. dot5 significantly outperforms t5 sequential training across all tasks, and achieves zero-shot sota in radiology nli with massive gains. we also conduct extensive analyses to identify the contribution from each model component and the benefits from scaling up the model size, and demonstrate direct evidence of domain-specific task knowledge learned from dot5's compositional transfer. limitations of this work include the challenge of drawing clear boundaries between domains and the necessarily incomplete exploration of hyperparameters and configurations. for example, general domain texts may contain biomedical or radiology sources, and our 'biomedical' nli evaluation set leans strongly clinical, introducing a degree of domain shift. investigation of the weighting of terms in the loss reveals the potential to improve performance through more exhaustive hyperparameter search-we emphasise that this was a proof-of-concept study and although dot5 performs favourably, zero-shot domain transfer could be further pushed, especially if only a single downstream task is required. the proposed nlgu method and subsequent self-finetuning was critical for improving downstream task performance. however, we observed an intermittent negative effect wherein the model would attempt to solve the nlu task when presented with an unusually long prompt. further work can be done to refine this approach. for example, the benefit of nlgu in resource-rich domains is unclear. as our focus is on domain transfer and we do not evaluate on general-domain tasks, we leave such experimentation to future study. finally, we acknowledge that it is non-trivial to apply our full framework to single-sentence/ paragraph classification tasks. while our most basic setup (compositional training of in-domain mlm and vanilla task training) can still be transferable to any task format, nlgu and selffinetuning would currently only work for tasks that involve pairs of texts. nonetheless, we believe dot5 proves to be a highly effective zeroshot domain transfer framework which will be beneficial to domain-specific applications beyond radiology and biomedicine.