Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks (2022)

Abstract: Natural language understanding (NLU) has made massive progress driven by large benchmarks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained models on the long tail? We conceptualize the long tail using macro-level dimensions (underrepresented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning research for NLU. Our analysis asks three questions: (i) Which long tail dimensions do transfer learning studies target? (ii) Which properties of adaptation methods help improve performance on the long tail? (iii) Which methodological gaps have greatest negative impact on long tail performance? Our answers highlight major avenues for future research in transfer learning for the long tail. Lastly, using our meta-analysis framework, we perform a case study comparing the performance of various adaptation methods on clinical narratives, which provides interesting insights that may enable us to make progress along these future avenues.

## introduction according to zipf's law, many micro-level language phenomena naturally occur infrequently and will be relegated to the long tail, except in cases of intentional over-sampling. moreover, the advantages of restricting community focus to a specific set of benchmark corpora and limitations in resources lead to portions of the macro-level space being under-explored, which can further cause certain micro-level phenomena to be under-represented. for example, since most popular coreference benchmarks focus on english narratives, they do not contain many instances of zero anaphora, a phenomenon quite common in other languages (e.g., japanese, chinese). in such situations, model performance on benchmark corpora may not be truly reflective of expected performance on micro-level long tail phenomena, raising questions about the ability of state-of-the-art models to generalize to the long tail. most benchmarks do not explicitly catalogue the list of micro-level language phenomena that are included or excluded in the sample, which makes it non-trivial to construct a list of long tail micro-level language phenomena. hence, we formalize an alternate conceptualization of the long tail: undersampled portions of the macro-level space that can be treated as proxies for long tail micro-level phenomena. these undersampled long tail macro-level dimensions highlight gaps and present potential new challenging directions for the field. therefore, periodically taking stock of research to identify long tail macro-level dimensions can help in highlighting opportunities for progress that have not yet been tackled. this idea has been gaining prominence recently; for example, @xcite survey languages studied by nlp papers, providing statistical support for the existence of a macro-level long tail of low-resource languages. in this work, our goal is to attempt to characterize the macro-level long tail in natural language understanding (nlu) and efforts that have tried to address it from research on transfer learning. large benchmarks have driven much of the recent methodological progress on nlu @xcite @xcite @xcite , but the generalization abilities of benchmark-trained models to the long tail have been unclear. in tandem, the nlp community has been successfully developing transfer learning methods to improve generalization of models trained on nlu benchmarks @xcite . the goal of transfer learning research is to tackle the macro-level long tail in nlu, leading to the question: how far has transfer learning addressed performance of benchmark models on the nlu long tail, and where do we still fall behind? probing further, we perform a qualitative meta-analysis of a representative sample of 100 papers on domain adaptation and transfer learning in nlu. we sample these papers based on citation counts and publication venues ( §2.1), and document seven facets for each paper such as tasks and domains studied, adaptation settings evaluated, and so on. ( §2.2). adaptation methods proposed (or applied) are documented using a hierarchical categorization described in §2.3, which we develop by extending the hierarchy from @xcite . with this information, our analysis focuses on three questions: the rest of the paper presents thorough answers to these questions, laying out avenues for future research on transfer learning that more effectively address the macro-level long tail in nlu. we also present a case study 1 to demonstrate how our meta-analysis framework can be use to systematically design experiments that provide insights that enable us to make progress along these avenues. ## case study: evaluating adaptation methods on clinical narratives finally, we attempt to demonstrate how our meta-analysis framework and observations can be used to systematically design case studies that can provide answers to the prevailing open questions laid out previously. as an example, we conduct a case study to evaluate the effectiveness of popularly used adaptation methods on high-expertise domains in an unsupervised adaptation setting, a burgeoning area of interest @xcite . specifically, our study focuses on the question: which method categories perform best for semantic sequence labeling tasks when transferring from news to clinical narratives, given an unsupervised setting (i.e., no labeled clinical data available)? we focus on two semantic sequence labeling tasks: entity extraction and event extraction. ## conclusion this work presents a qualitative meta-analysis of 100 representative papers on domain adaptation and transfer learning in nlu, with the aim of understanding performance of adaptation methods on the long tail. through this analysis, we assess current trends and highlight methodological gaps that we consider to be major avenues for future research in transfer learning for the long tail. we observe that long tail coverage in current research is far from comprehensive, and identify two properties of adaptation methods that may improve long tail performance, but have been under-explored: (i) incorporating source-target distance, and (ii) incorporating nuanced domain variation. additionally, we identify three major gaps that must be addressed to improve long tail performance: (i) combining adaptation methods, (ii) incorporating external knowledge, and (iii) application to data-scarce adaptation settings. finally, we demonstrate the utility of our meta-analysis framework and share observations in guiding the design of systematic meta-experiments to address prevailing open questions by conducting a systematic evaluation of popular adaptation methods for high-expertise domains in a data-scarce setting. this case study reveals interesting insights about the adaptation methods evaluated and shows that significant progress can be made towards developing a better understanding of adaptation for the long tail by conducting such experiments.