Modular Domain Adaptation (2022)

Abstract: Off-the-shelf models are widely used by computational social science researchers to measure properties of text, such as sentiment. However, without access to source data it is difficult to account for domain shift, which represents a threat to validity. Here, we treat domain adaptation as a modular process that involves separate model producers and model consumers, and show how they can independently cooperate to facilitate more accurate measurements of text. We introduce two lightweight techniques for this scenario, and demonstrate that they reliably increase out-of-domain accuracy on four multi-domain text classification datasets when used with linear and contextual embedding models. We conclude with recommendations for model producers and consumers, and release models and replication code to accompany this paper.

## introduction machine learning models for tasks like sentiment analysis and hate speech detection are becoming increasingly ubiquitous as off-the-shelf tools, including as commercial packages or cloud-based apis. among other applications, these models are widely used by computational social scientists to obtain standardized measurements of various document properties at scale. however, the problem of domain shift represents a threat to validity, one which is difficult for practitioners to overcome, especially without access to source data-which may be unavailable for reasons of privacy, copyright, or commercial interests. in this paper, we propose to treat domain adaptation as a modular process involving both model producers and model consumers, and show how both parties can independently cooperate to produce more reliable measurements. although this framework applies to any application involving independent model producers and consumers, we focus here on text-based instruments, including both lexicons and supervised text classification models. using multiple datasets and baselines, we show that model consumers can obtain more accurate results by using models designed to be lightly adapted, and that model producers can facilitate such adaptation, even without providing access to source data, using what we call anticipatory domain adaptation (see figure 1 ). we introduce two techniques under this new paradigm: domain-specific bias (dsbias) and domain-specific normalization (dsnorm). these methods enable model consumers to incorporate information from their domain of interest-without additional training or hyperparameter tuning-and provide reliably better out-of-domain accuracy for both linear and contextual embedding classifiers. in summary, this paper makes the following contributions: ## experiments in this section we systematically evaluate the performance of both underlying models in conjunction with all available techniques in section §3, to quantitatively evaluate their performance, and to derive best practices as advice to practitioners when applying them to real data under various settings. for simplicity, we use accuracy as the primary metric of evaluation in all our experiments. ## discussion and recommendations a key idea of this paper is that domain adaptation should not be something that only model consumers have to confront. rather, we should think of domain adaptation as a modular, collaborative process, in which model producers should anticipate that model consumers will want to apply models to new domains. ideally, model producers would also make training data available to model consumers, so as to facilitate domain adaptation. for settings in which this is not possible, we have presented two techniques (dsbias and dsnorm) which improved performance for both logistic regression and contextual embedding models, and we encourage the development of additional techniques. although it is still useful for model producers to report performance in the training domain as part of model documentation @xcite , model consumers should not rely on such estimates for offthe-shelf models, given the expected performance drop across domains @xcite ; see also appendix c). rather, it is essential to have sufficient labeled data in the application domain to be able to estimate performance, in addition to any labeled data to be used for adaptation, and this should be budgeted for when planning annotations @xcite . for specific applications, model consumers may also care about metrics beyond accuracy, and should evaluate models based on what is most relevant. in addition, these ideas could be fruitfully combined with techniques for lexicon expansion, to account for terms which were not present in the original domain(s) @xcite . lexicons such as liwc have an enduring popularity, in part because of their ease of use. as the results above demonstrate, however, simple logistic regression models can do as well (in terms of classification accuracy). contextual embedding models derived from the same data are considerably more accurate, and need not be any more difficult for practitioners to apply. thus, we encourage css researchers to produce and share such models, even if the raw data itself cannot be shared. ## conclusion using off-the-shelf text classification models for computational social science requires careful thought regarding domain shift. in this paper, we approach this as a modular process in which model producers can apply techniques of anticipatory domain adaptation to facilitate adaptation by model consumers. we demonstrate that using domainspecific bias (dsbias) and domain-specific normalization (dsnorm) produces a reliable performance boost for the model consumers, and that this applies to both linear and contextual embedding models. finally, for cases where accuracy is more important than interpretability, we demonstrate the superior out-of-domain performance of contextual embedding models when compared to linear models, even without additional fine-tuning, and encourage model producers to make multiple types of models available.