Word Matters: What Influences Domain Adaptation in Summarization? (2024)

Abstract: Domain adaptation aims to enable Large Language Models (LLMs) to generalize domain datasets unseen effectively during the training phase. However, factors such as the size of the model parameters and the scale of training data are general influencers and do not reflect the nuances of domain adaptation performance. This paper investigates the fine-grained factors affecting domain adaptation performance, analyzing the specific impact of ‘words’ in training data on summarization tasks. We propose quantifying dataset learning difficulty as the learning difficulty of generative summarization, which is determined by two indicators: word-based compression rate and abstraction level. Our experiments conclude that, when considering dataset learning difficulty, the cross-domain overlap and the performance gain in summarization tasks exhibit an approximate linear relationship, which is not directly related to the number of words. Based on this finding, predicting a model’s performance on unknown domain datasets is possible without undergoing training. Source code and scripts are available at https://github.com/li-aolong/Word-Matters.

## introduction with the continuous development of large language models (llms), remarkable capabilities have been demonstrated in knowledge comprehension @xcite , logical reasoning @xcite , problem-solving @xcite , and other aspects @xcite . as a result, llms have been widely applied to various summarization tasks on different domains to improve productivity like law @xcite , medicine @xcite , finance @xcite and so on, including both natural and social science study @xcite . however, when llms are applied to specific domains, it often necessitates the selection of corresponding domain-specific knowledge bases for training @xcite @xcite . this results in a limitation where a model trained in one domain struggles to be effectively applied in others @xcite , leading to a waste of resources. this constraint arises from the disparity in the distribution between the training data and the target domain data @xcite . in light of this limitation, effective methods must be taken to fix the gap and then enhance the model's adaptability and efficiency in summarization tasks. domain adaptation aims to train a model from multiple source domains, enabling it to generalize well to unseen domains @xcite . consequently, enhancing domain adaptation performance is a key objective for large-scale models in improving downstream tasks @xcite . it is worthwhile to explore which factors can affect the domain adaptation performance @xcite . @xcite proposes that metrics based on nonlinear or non-contiguous tokens are crucial to a model demonstrating emergent abilities and that rouge-l-sum shows sharper variations. this has inspired us to consider the performance changes of models in domain adaptation from the perspective of more granular units. tokens typically do not possess complete semantics, whereas words are the basic language units with specific meanings or functions. therefore, we consider exploring the impact on model performance in domain adaptation from the perspective of words. summarization tasks involve generating concise texts that encapsulate the main components of longer documents, considering factors such as coherence, information diversity, and coverage scope @xcite . this differs from other downstream tasks like machine translation @xcite and classification @xcite . @xcite note that reducing summary extractors' size or compression ratio can lead to losing vital content, features, concepts, and other significant information. therefore, we explore how the degree of information extraction between input documents and target summaries impacts domain adaptation performance in summarization tasks. this paper investigates how words impact the domain adaptation of summary tasks. we first introduce two indicators, compression rate, and abstraction level, to quantify the learning difficulty of datasets, thereby more accurately reflecting the performance gain of models. then, we identify two key aspects affecting domain adaptation: crossdomain overlap and word count, hypothesizing a linear relationship between them and model performance. experiments are conducted with models of various sizes on summarization datasets from four domains. the results indicate that the crossdomain overlap exhibits an approximately linear relationship with performance gain when considering dataset learning difficulty. in contrast, word count shows no significant correlation. based on this linear relationship, it is possible to predict model performance without undergoing training by using the cross-domain overlap calculated from the dataset. our contributions can be summarized as follows: ## related work domain adaptation (da) has emerged as a crucial methodology for enhancing model performance across varying domains @xcite . da aims to enhance the performance of llms in a target domain, where annotated data may be scarce or absent, by leveraging knowledge from a related domain with a sufficient amount of labeled data @xcite . many methods have been developed to tackle the out-of-domain adaptation issue @xcite @xcite @xcite . there are three pivotal strategies related to our work: (1) continual pre-training, (2) alignment of distributions, and (3) adaptation tuning. continual pre-training continual pre-training uses similar training objectives as continual selfsupervised training to update pre-trained models with new data instead of retraining from scratch @xcite . continual pre-training is studied for domain adaptation where the new dataset comes from a new domain, which is referred to as continual domain-adaptive pre-training (da-training) @xcite @xcite . dap-training methods can achieve better results by training llms with a large unlabeled domain corpus before endtask fine-tuning @xcite @xcite . however, the effectiveness of this method is contingent upon the relevance of the pre-trained llms to the target domain and requires substantial domainspecific data to achieve optimal performance. ## what and how does word influence domain adaptation? in this section, we explore how words can affect aspects of domain adaptation and their impact. we first hypothesize that datasets of varying learning difficulties affect model performance and investigate the influence of words on the learning difficulty of target domain datasets. we propose two indicators to reflect the learning difficulty of datasets: compression ratio and abstraction level. secondly, from the perspective of words, we propose two aspects that could affect domain adaptation: cross-domain overlap and word count. finally, we hypothesize a linear relationship between these aspects and domain adaptation performance based on dataset learning difficulty. this hypothesis is tested in subsequent experiments. ## experiments we use four summarization datasets, cnn/daily mail @xcite , pubmed @xcite , samsum @xcite and wiki-how @xcite , each originating from the news, science, conversation, and general domains respectively. due to the significant differences in the number of samples across different datasets, we sample 35,000 samples from the cnndm, pubmed, and wikihow datasets for the training set while retaining the entire samsum dataset. we sample 500 test samples from all the test sets of these datasets. the detailed statistical data of the datasets are shown in to verify the above two hypotheses, we configure different experimental setups for cross-domain overlap and word count. all experiments are based on the bloom @xcite and llama2 @xcite series of models. due to resource constraints, different experiments use models of various sizes. the prompts used to train the llama2 and bloom models are presented in appendix a. ## results and analysis we analyze the results from different perspectives. both single-domain and multi-domain experiment results supported our hypothesis that cross-domain overlap is linearly correlated with performance. due to this finding, we could make a quantitative prediction about the transferred result of certain given training and evaluating datasets. on the other hand, we found that word count is unrelated to domain adaptation, which further emphasizes the significance of choosing high-quality data rather than a large amount of data. to solidify the universality of our findings, we conduct a prediction study and it turns out that our hypothesis can successfully predict domain adaptation performance with cross-domain overlap. there is only a slight gap between our prediction data and observed data. ## conclusion we investigate the impact of words on domain adaptation performance in summarization tasks. we propose two indicators to represent the learning difficulty from a dataset and introduce a performance evaluation method based on learning difficulty. we find that word overlap is an essential factor affecting domain adaptation and exhibits a linear correlation with model performance. however, the influence of word count on domain adaptation does not show a regular pattern. we will investigate this phenomenon further in future work. furthermore, by predicting the performance of new domain data based on its cross-domain overlap with existing domains, it becomes possible to preemptively assess the model's suitability for specific domains without the need for extensive retraining or fine-tuning. this predictive capability can significantly streamline the adaptation of language models to new domains and save many resources, finally improving their practical utility in real-world applications.