M ulti CQA : Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale (2020)

Abstract: We study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source domains from community question answering forums in English. We investigate the model performances on nine benchmarks of answer selection and question similarity tasks, and show that all 140 models transfer surprisingly well, where the large majority of models substantially outperforms common IR baselines. We also demonstrate that considering a broad selection of source domains is crucial for obtaining the best zero-shot transfer performances, which contrasts the standard procedure that merely relies on the largest and most similar domains. In addition, we extensively study how to best combine multiple source domains. We propose to incorporate self-supervised with supervised multi-task learning on all available source domains. Our best zero-shot transfer model considerably outperforms in-domain BERT and the previous state of the art on six benchmarks. Fine-tuning of our model with in-domain data results in additional large gains and achieves the new state of the art on all nine benchmarks.

## introduction semantic matching of two text sequences is crucial among a wide range of nlp problems, such as question answering @xcite or semantic textual similarity @xcite . due to the ubiquity of applications, it is crucial to study how to obtain re-usable text matching models that transfer well to unseen domains or tasks. zero-shot transfer of text matching models is particularly challenging in setups of non-factoid answer selection @xcite @xcite and question similarity @xcite . these tasks compare questions and answers, or two potentially related questions in community question answering (cqa) forums, faq pages, and general collections of text passages. in contrast to other text matching tasks in nlp, they compare texts of different lengths-e.g., answers can be long explanations or descriptions-and often deal with expert domains. this makes it difficult to transfer models across domains @xcite , and to apply common approaches such as universal sentence embeddings without further domain or task adaptations @xcite . non-factoid answer selection and question similarity are also particularly promising to study zeroshot transfer. reasons are that (1) there exist a large number of domains, and (2) in-domain training data is often scarce. previous work proposed domain adaptation techniques @xcite , training with unlabeled data @xcite , and shallow architectures @xcite . however, these approaches result in entirely separate models that are specialized to individual target domains. one model that is re-usable and targets zero-shot transfer in similar settings is the question-answer encoder of @xcite , which has recently been evaluated in cross-domain settings for efficient answer sentence retrieval @xcite . however, they do not study zero-shot transfer with a large number of source domains, and they do not assess how to best combine them. in this work, we address these limitations and are-to the best of our knowledge-the first to study the zero-shot transfer capabilities of re-usable text matching models with a large number of source domains in these challenging setups. in the first part, we investigate the zero-shot transfer capabilities of 140 domain-specific text matching models to nine benchmark datasets. by leveraging self-supervised training signals of ques-tion title-body pairs, we analyze a large number of models specialized on diverse domains. we utilize the training method provided by @xcite and train adapter modules @xcite within bert @xcite for each of the 140 english stack-exchange forums. adapters considerably reduce storage requirements by training only a small number of additional parameters while keeping the pre-trained bert weights fixed. in our extensive analysis, we show that our approach for zero-shot transfer is extremely effective-on six benchmarks all 140 models outperform common ir baselines. most importantly, we revisit and analyze the traditional strategy of leveraging large data sets from intuitively similar domains to train models for zeroshot transfer. we establish that neither training data size nor domain similarity are suitable for predicting the best models, stressing the need for more elaborate strategies to identify suitable training tasks. this also demonstrates that considering a broad selection of source domains is crucial, which contrasts the standard practice of merely relying on the most similar or largest ones. in the second part of this work, we study how to best combine multiple source domains with multitask learning and adapterfusion @xcite . our analysis reveals that both approaches are not affected by catastrophic interference across training sets. in particular, our combination of all available source domains-despite the large data imbalances, see figure 1 -is the most effective and outperforms the respective best of 140 singledomain models on six out of nine bechmarks. finally, we combine unlabeled with labeled data for training in a self-supervised and supervised fashion, which considerably improves the zero-shot transfer performances in 16 out of 18 cases. our best model substantially outperforms the in-domain bert and roberta @xcite models, as well as the previous state of the art on six benchmarks, which demonstrates its versatility across tasks and domains. we also show that our model is an effective initialization for in-domain fine-tuning, which results in large gains and achieves state-of-the-art results on all nine benchmarks. our source code and the weights of our best multi-task model is publicly available. foot_0 additionally, all 140 source domain adapters are available at adapterhub.ml @xcite . ## related work the predominant method for text matching tasks such as non-factoid answer selection and question similarity is to train a neural architecture on a large quantity of labeled in-domain data. this includes cnn and lstm models with attention @xcite @xcite , compare-aggregate approaches @xcite , and, more recently, transformer-based models @xcite @xcite . fine-tuning of large pre-trained transformers such as bert @xcite and roberta @xcite currently achieves stateof-the-art performances on many related benchmarks @xcite @xcite . however, realistic scenarios often do not provide enough labeled data for supervised in-domain training. thus, different recent work has focused on improving model performances in small data scenarios. @xcite use adversarial domain adaptation for duplicate question detection. @xcite adapt the combination of different sentence embeddings to individual target domains. @xcite use weakly supervised training, self-supervised training methods, and question generation. similar approaches were also explored in ad-hoc retrieval @xcite @xcite . a crucial limitation of these approaches is that they result in entirely separate models for each dataset and are thus not re-usable. in this work, we therefore explore the zero-shot transfer capabilities of models, to understand how well they generalize to unseen settings. previous work of @xcite investigates this on a smaller scale. they propose use-qa, a sentence encoder for comparing questions and answers, and achieve promising zero-shot results in retrieval tasks. however, it is unclear how this model compares to the zero-shot performances of models trained on several different source domains and how best to combine the data from multiple domains. other work addresses the generalization of models over several domains in different settings, e.g., for machine reading comprehension @xcite . more related to our work, @xcite propose a new evaluation suite with eight datasets for retrieval-based qa, in which they also study the effectiveness of use-qa. in contrast to them, our work (1) deals with re-ranking setups and uses cross-encoders, which is different to their bi-encoder scenario for retrieval; ## zero-shot transfer from 140 domains in this section, we study the zero-shot transfer performances of all models ( §4.1) and investigate whether domain similarity and training data size are suitable for predicting the best models ( §4.2). ## zero-shot transfer from combinations of multiple domains we now investigate how to best combine multiple source domains for zero-shot transfer. we denote our models as multicqa. ## analysis we manually inspect 50 instances of insuranceqa and askubuntu for which our zero-shot transfer model multicqa rba-lg selects a wrong answer or an unrelated question. we find that the texts are always on-topic, i.e., many aspects of the question are included in the selected answers (insur-anceqa) or in the potentially similar questions (askubuntu). this includes keywords, phrases (often paraphrased), names, version numbers, etc. the most common source of error is that an important aspect of the question appears to be ignored or is (likely) not understood by the model. for instance, many aspects of the question might be mentioned in a potentially similar question on askubuntu, but in the wrong context. query question: passing parameters to the installer for 14.04? the installer for 14.04 gave me no chance (that i took notice of) to pass parameters [...] most similar (multicqa rba-lg ): which key combination would allow me to pass parameters to kernel? during boot i want to pass some parameters like the runlevel , nomodeset to kernel during the booting process [...] ground truth: how can i customize the ubuntu installer? i would like to know how can i customize the ubuntu installer not customize ubuntu , i just want to modify the installer [...] ## conclusion we studied the zero-shot transfer of text matching models on a massive scale, with 140 different source domains and nine benchmark datasets of non-factoid answer selection and question similarity tasks. by investigating such a large number of models, we provided an extensive comparison and fair baselines to combination methods, and were able to extensively analyze a large sample size. we have shown that (1) bert models trained in a self-supervised manner on cqa forum data transfer well to all our benchmarks, even across distant domains; (2) training data size and domain similarity are not suitable for predicting the zeroshot transfer performances, revealing that a broad selection of source domains is crucial; (3) our multicqa approach that combines self-supervised and supervised training data across a large set of source domains outperforms many in-domain baselines and achieves state-of-the-art zero-shot performances on six benchmarks; (4) fine-tuning multicqa rba-lg in-domain further improves the performances and achieves state-of-the-art results on all nine benchmarks. we clearly demonstrated the effectiveness and the relevance of zero-shot transfer in many realistic scenarios and believe that our work lays foundations for a wide range of research questions. for instance, combining our approach with additional pre-training objectives such as the inverse cloze task @xcite could substantially increase the amount of training data for the large quantity of smaller forums. researchers could also use our 140 domain-specific adapters and investigate further combination techniques to make them even more broadly applicable.