BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification (2023)

Abstract: While performance of many text classification tasks has been recently improved due to Pretrained Language Models (PLMs), in this paper we show that they still suffer from a performance gap when the underlying distribution of topics changes. For example, a genre classifier trained on political topics often fails when tested on documents in the same genre, but about sport or medicine. In this work, we quantify this phenomenon empirically with a large corpus and a large set of topics. Thus, we verify that domain transfer remains challenging both for classic PLMs, such as BERT, and for modern large models (LLMs), such as GPT. We develop a data augmentation approach by generating texts in any desired genre and on any desired topic, even when there are no documents in the training corpus that are both in that particular genre and on that particular topic. When we augment the training dataset with the topically-controlled synthetic texts, F1 improves up to 50% for some topics, approaching on-topic training, while showing no or next to no improvement for other topics. While our empirical results focus on genre classification, our methodology is applicable to other classification tasks such as gender, authorship, or sentiment classification.

## introduction automatic text classification is a critical task in natural language processing, enabling proper understanding, summarization, archiving, and retrieval of documents across various domains, such as legal and medical. this task has been greatly improved due to pre-trained language models such as bert @xcite , t5 @xcite or gpts @xcite . to achieve true artificial general intelligence (agi), it is essential that trained computer models can recognize various document categories across different domains. however it has been noticed @xcite that while in general plms are more robust than previous models, they still suffer from spurious domain-specific clues. while all our methods proposed here apply to other non-topical text classification tasks such as sentiment or authorship identification, in this particular work we have taken a thorough look at document genre classifiers: distinguishing between different styles (genres) of texts, such as academic articles, experimental protocols, regulatory documents, and patient leaflets @xcite . people can easily recognize document genres from just a few examples even if those examples are from a different domain @xcite . text classification research often contrasts the properties of topic vs. those of style @xcite . however, this contrast is difficult to maintain, as the training sets in most corpora for style or genre prediction are biased with respect to topics specific to individual styles or genres, so that classifiers do not transfer across corpora in case of variation between their topics. for example, a model identifying faqs can learn to pay attention to such keywords as hurricane and tax advice in case these topics are common for faqs in a specific training corpus @xcite . so far, this cross-influence of topics and styles has not been studied in the context of plms such as bert @xcite , t5 @xcite or gpts @xcite . there has also been no quantification of the gap in transferring genre/style classifiers to new domains. for instance, no study has yet assessed the performance degradation when a classifier is trained on political topics but tested on texts about sports or medicine. in light of the aforementioned challenges, our study offers the following novel contributions foot_0 : chatgpt we were able to confirm that even a much larger language model can still suffer from a domain transfer gap. ## related studies and baselines there have been studies that looked at impact of out-of-domain training data on plm-based classifiers. in particular, @xcite noticed that while in general plms are more robust than previous models, they still suffer from spurious clues. however, they tested the transfer gap only on a few hand-picked datasets with similar tasks but different data distributions (e.g. sentiment analysis trained on book reviews applied to movie reviews), while here we are presenting an original methodology based on a neural topic model to investigate domain transfer between a wide variety of topics. also, none of the prior works looked at domain transfer for genre/style classification tasks which we do here. within the broader context of domain transfer, genre classification holds a unique position. automatic genre classification has been recognised as an important task since the 1990s @xcite . the effect of topical biases has been estimated empirically by considering the reduction in performance of genre classifiers across topics in the new york times corpus @xcite . several studies have also demonstrated the success of plms with respect to the genre classification tasks @xcite . however, there have been no studies of topical biases for these models. the split between topics and styles has been studied for a related task, including disentangled representation @xcite and other methods of topic-style decomposition @xcite . however, our study focuses on the numerical estimates of the topic transfer gap on large samples diverse in topics and in genres. a related research area concerns the use of causal models for interpreting the biases of neural predictions, for example, with respect to gender @xcite . there have been studies to investigate biases in neural models by adding counter-factuals @xcite . it has been noted that well-established data augmentation (da) methods in domains such as computer vision and speech recognition @xcite @xcite , relying on simple transformations of existing samples, cannot be easily applied to natural text since they can lead to syntactic and semantic distortions. for a survey of da approaches for various natural language processing tasks we refer a reader to @xcite . the survey mentions several studies showing that da is generally much less beneficial when applied to out-of-domain data (as studied here), likely because "the distribution of augmented data can substantially differ from the original data." while only a few of the surveyed works involved plms, the survey points out that plms have made many previously useful da techniques obsolete since fine-tuned plm-based classifiers already achieve high performance, as they have been pre-trained on large and diverse corpora. for those reasons, we decided not to contrast our approach with any of the classical pre-plm domain transfer techniques, such as blitzer et al. (2007) or daumé iii et al. (2010). while up to our knowledge, none of the prior works has specifically looked into the domain transfer gap for genre (or style) classification, it is still worth to note several closely related works, some of them included in feng et al. (2021) survey that involved plms not only as classifiers but also as generators for augmented data. this includes kumar et al. (2020) who looked at sentiment/intent/question classification, lee et al. (2021) who targeted under-represented categories, edwards et al. (2021) who focused on selecting the seeds examples to train augmentation generation in the context of few-shot classification, and yang et al. ( 2020 ) focused on lowresource in commonsense reasoning. since the augmentation approach tried in those works is based on straightforward training (fine-tuning) a plm-based text generator using the existing data (without exercising any topical control), we include the results from this general approach in "aug baseline" column in addition to the baseline that does not attempt any augmentation ("offtopic" column in ## methodology our study builds upon prior investigations into domain biases in text classification @xcite , which largely depended on a limited set of hand-selected datasets with analogous tasks but varying distributions. we present a comprehensive methodology to assess and mitigate the domain transfer gap. the main idea is to simulate the situation when a classifier is trained on documents that lack a topic, e.g. medicine, and then to test it on the documents where such topic is well represented. this performance is contrasted with the situation when the classifier is initially trained on the documents where this topic is represented well. while our empirical results focus on genre classification, our methodology is directly applicable to other classification tasks such as gender, authorship, or sentiment classification. we train two classes of models: figure 1 illustrates the overall workflow for our experiments. ## experiments the most time-consuming part of our experiments were fine-tuning the generators (t5) and the classifiers at the cost of roughly 6000 hours of nvidia geforce rtx 2080. ## conclusions we have demonstrated a severe degradation in a plm-based document classifier when trained on one topic, such as politics, and tested on another, like healthcare. rather than following the prior empirical studies on the impact of domain transfer that involved only a few hand-picked datasets with similar tasks but somewhat different data distributions, we have developed a methodology based on a neural topic model to assess the domain transfer gap between a wide variety of topics. while our empirical results focus on genre classification, our methodology is applicable to other classification tasks such as gender, authorship, or sentiment classification. we have also shown that the topic transfer gap can be mitigated by means of proper topic control while generating additional training documents (augmentation). as a result of our approach, a model to predict a non-topical category (genres in the case here) can be trained on the documents in one topic (e.g. politics) and applied to another (e.g. healthcare) even when there are no healthcare-related documents in the training corpus. we have also created a large corpus with natural genre annotation and a very general/diverse topic model. both can be used in follow-up studies. still, our study has certain limitations. the degree of improvements from augmentation is not uniform. for some topics we obtain much better results than for others, while occasionally the performance on the augmented set is even lower than on the original off-topic training set. this is likely to be related to the high degree of correlation between the topics and genres, for example, the lack of texts on the topic of law in genres other than news reporting in our corpus, thus leading to less successful attempts to generate discussions, academic articles or advice texts on this topic. we need to find better ways to improve offtopic generation when it makes no positive impact on the accuracy of classification of on-topic test texts, possibly by using very large language models. nevertheless, through a qualitative exploratory study with chatgpt we were able to confirm that even such larger language models still suffer from the domain transfer gap. even while our approach does not solve this very challenging domain transfer problem completely, it suggests a direction in which a small but productive step can be made. larger pre-trained language models can be tried in future such as gpt-4, for both generation and classification. also, larger training sets can be explored, as well as in "few-shot" settings. a number of approaches improving the quality of generated text, e.g. those based on generative adversarial networks @xcite or meta learning @xcite can be explored, as well as various methods to control the quality and topical fit of the generated texts.