FETA : A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue (2022)

Abstract: Task transfer, transferring knowledge contained in related tasks, holds the promise of reducing the quantity of labeled data required to fine-tune language models. Dialogue understanding encompasses many diverse tasks, yet task transfer has not been thoroughly studied in conversational AI. This work explores conversational task transfer by introducing FETA: a benchmark for FEw-sample TAsk transfer in open-domain dialogue.FETA contains two underlying sets of conversations upon which there are 10 and 7 tasks annotated, enabling the study of intra-dataset task transfer; task transfer without domain adaptation. We utilize three popular language models and three learning algorithms to analyze the transferability between 132 source-target task pairs and create a baseline for future work.We run experiments in the single- and multi-source settings and report valuable findings, e.g., most performance trends are model-specific, and span extraction and multiple-choice tasks benefit the most from task transfer.In addition to task transfer, FETA can be a valuable resource for future research into the efficiency and generalizability of pre-training datasets and model architectures, as well as for learning settings such as continual and multitask learning.

## introduction improving sample efficiency through transfer learning has been a long-standing challenge in the machine learning and natural language processing communities @xcite . dialogue data requires multiple cohesive turns with consistent speaker personalities @xcite , creating a challenge for data collection and motivating the development of techniques that improve sample efficiency in conversational ai @xcite . furthermore, dialogue understanding tasks require a shared knowledge of semantics, pragmatics, human behavior, and commonsense, making dialogue an area of study that can benefit greatly from a deeper understanding of transfer learning. two essential transfer learning settings, namely domain adaptation and task transfer, have been studied on language tasks @xcite . while domain adaptation has been studied in taskoriented dialogue @xcite , task transfer has been studied with less rigor in conversational ai. prior studies of task transfer in dialogue consider only 2-4 tasks, focus on multitask learning, and do not compare learning algorithms @xcite . prior studies have focused on cross-dataset task transfer, gathering tasks annotated on disjoint datasets @xcite , but this can lead to improvements in domain adaptation being confounded as improvements in task transfer. a precise study of task transfer should be on a sin-gle data source in an intra-dataset transfer setting, as in @xcite . additionally, previous studies focus on learning algorithms and use only a single language model architecture @xcite @xcite , which may lead to a narrow understanding. to the best of our knowledge, this is the first rigorous study on task transfer in dialogue and the most extensive intra-dataset task transfer study in nlp. in this work, we create feta, a benchmark for few-sample task transfer for language understanding in open-domain dialogue with 17 total tasks. feta datasets cover a variety of properties (dyadic vs. multi-party, anonymized vs. recurring speaker, varying dialogue lengths) and task types (utterance-level classification, dialogue-level classification, span extraction, multiple-choice), and maintain a wide variety of data quantities. we study task transfer on feta by comparing three task transfer algorithms and three commonly used language models in single-source and multisource settings. figure 1 illustrates some results in the single-source setting. for example, we find that dialogue reasoning span extraction benefits from nearly all source tasks. on the other hand, adversarial response selection and emotion recognition improve the performance of many target tasks when utilized as a source task. in this study, we find that: (i) trends are largely model-dependent, a finding that previous works have not discussed. (ii) out of all task types, span extraction tasks gain the most as a target, especially with few samples. (iii) adding source tasks does not uniformly improve over a single source task, motivating a better understanding of the complex relationship between source and target tasks. feta provides a resource for various future studies, e.g., on the generalizability of model architectures, and pre-training datasets that enable efficient transfer. in addition to task transfer, feta can also facilitate the study of continual and multitask learning. in summary, our main contributions are: more recently, dialoglue @xcite and raddle @xcite ) study domain adaptation for language understanding tasks in taskoriented dialogue. shuster et al. (2020) focuses on multitasking in dialogue response generation across multiple datasets. similar to this work, pruksachatkun et al. (2020) study task transfer, although they study cross-dataset task transfer in general nlp tasks. lourie et al. (2021) also study task transfer, but they focus on the t5 model and a suite of commonsenseqa datasets. task transfer in dialogue task transfer has been applied in task-oriented dialogue (tod) settings but never rigorously studied. for example, hosseini-asl et al. (2020) and lin et al. (2020) develop multitask models to perform 2-4 tod tasks but do not aim to analyze the efficiency of models or learning algorithms for task transfer. intra-dataset task transfer intra-dataset task transfer has been studied in computer vision applications @xcite , but to our best knowledge it has never been studied in nlp. ## feta in this section, we briefly define intra-dataset task transfer, the problem setting of feta. then, we introduce feta, our benchmark for few-sample task transfer in open-domain dialogue. finally, we define the metrics we use to evaluate models and learning algorithms on feta. ## experiment setup to study task transfer on feta, we run extensive experimentation. we utilize three task transfer algorithms: pre-train/fine-tune, multitask, and multitask/fine-tune, as described in section 4. to draw broad conclusions about the performance of each learning algorithm, we utilize pretrained language models with three different architectures: encoder-only (bert) @xcite , decoder-only (gpt-2) @xcite , and encoder-decoder (t5) @xcite . implementation details, including hyperparameters and prompts, can be found in appendix b. a complete experiment for a single target task, t , is as follows: first, we directly fine-tune on t to get the baseline score. then, for each source task, s, we take the model pre-trained on s and fine-tune on t . next, we jointly train on s and t together. finally, we fine-tune the jointly trained model on t . feta datasets have 10 and 7 tasks, giving 90 + 42 = 132 unique source-target task pairs. our experiments include three learning algorithms, three models, and we run each experiment with 5 random seeds. in total, we run 132 × 3 × 3 × 5 = 5940 transfer experiments, and 17×3×5 = 255 baseline experiments leading to 6195 trained models. in addition to the single-source setting described above, we also consider a subset of tasks to study in the multi-source setting, where multiple tasks are simultaneously used as source tasks to transfer to a single target task (6.2). for our experiments, we select two target tasks from each dataset that benefit the most from task transfer, and we use the three source tasks that transferred best onto those targets. ## conclusion we introduce feta, a comprehensive benchmark for evaluating language models and task transfer learning algorithms in open-domain dialogue with few samples. through extensive experimentation, we find new and non-intuitive insights on the mechanisms of transfer learning. in particular, we find that most trends are model-specific, and we strongly encourage researchers to consider multiple model architectures before attempting to draw broad conclusions on transfer learning. it is our hope that feta enables further research not only in task transfer, but also in other learning settings, and in the generalizability and efficiency of model architectures and pre-training datasets.