A dapt E val: Evaluating Large Language Models on Domain Adaptation for Text Summarization (2024)

Abstract: Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.

## introduction large language models (llm) have achieved remarkable improvements on a wide range of natural language processing tasks, including abstractive text summarization, the task of generating an abridged version of the most relevant information in a document @xcite . recent works study the domain adaptation abilities of llms on the summarization task. however, the research is still limited to a single domain, such as news articles @xcite or clinical reports @xcite . we argue that there is a lack of research across domains to better understand the abilities of these models to adapt to different targets. in this paper, we assess the domain adaptation abilities of 11 models, including conventional encoder-decoder models and a wide range of llms in various parameter sizes, on the summarization task. in particular, we experiment with fine-tuning and in-context learning (icl) settings and evaluate their performance across various domains (i.e. governmental, medical, and scientific), reporting scores on a collection of automatic-rouge @xcite and bertscore @xcite -and domain adaptation metrics. the latter includes domain vocabulary overlap @xcite , and our adaptations of g-eval @xcite and token distribution shift @xcite to the task. the experimental results show the abilities of llms to adapt to the domain in the icl setting. in particular, small models with 7b parameters achieve comparable performance to their larger counterparts with only two learning examples. however, g-eval highlights the difficulty of adapting to the medical domain. while the fine-tuned models achieve the best performance in terms of automatic scores, their adaptation to the domain vocabulary is inferior to the icl setting. finally, we release the domain benchmark and evaluation metrics as the first domain adaptation evaluation suite (adapteval) to facilitate the evaluation of models and foster further research on this task. foot_0 ## domain adaptation task we assess the performance of 11 models across domains in both fine-tuning foot_4 and icl settings. ## conclusion we evaluate the domain adaptation abilities of large language models across scientific, medical, and governmental domains using a set of adapted evaluation metrics. additionally, we release adapteval, an evaluation suite that facilitates the analysis of domain adaptation. our experiments show that smaller llms exhibit domain-shift challenges, but they are able to achieve comparable performance to larger llms when provided with only two learning examples. in contrast, fine-tuning does not have a significant impact on the vocabulary domain, but only on stylistic tokens. overall, the g-eval scores indicate that the medical domain is challenging for these models. we expect our work to encourage and facilitate further research on domain adaptation with llms across domains. we plan to continue this research in future work.