Domain Adapted Text Summarization with Self-Generated Guidelines (2025)

Abstract: Text summarization systems face significant adaptation costs when deployed across diverse domains, requiring expensive few-shot learning or manual prompt engineering. We propose a cost-effective domain adaptation framework that generates reusable summarization guidelines using only two reference summaries and three LLM inferences. Our approach works by having the model compare its own generated summaries against domain specific reference summaries in a one time preparation step that derives concise natural language guidelines that capture the summarization patterns of the target domain. These guidelines are then appended to the summarization prompt to adapt the LLM to the target domain at a minimal cost. We evaluate our method across diverse model sizes on three distinct summarization domains: Lawsuits, ArXiv papers, and Patents. Automatic metrics show that guideline-based adaptation achieves comparable or superior performance compared to in-context learning and zero-shot baselines. An LLM preference evaluation using the latest models shows that summaries generated using such guidelines are superior to the zero-shot or in-context learning summarization prompts. Our method enables efficient domain adaptation of text summarizer LLMs with a minimal resource overhead, making specialized summarization particularly accessible for agentic systems that require to process heterogeneous texts in enterprise environments.

## introduction text summarization modules are integral to many modern agentic systems @xcite as they enable agents to process larger volumes of information efficiently by condensing content within the limited input context. this allows agents to iteratively refine their understanding ## related work within the past couple of years, it has become increasingly apparent that llms outperform specialized seq2seq models at summarization tasks @xcite . a natural benefit of using llms for text generation tasks such as summarization is their long-context support and the ability to perform tasks in a manner that is more adapted to the specific use case through instruction prompt specifications. the search for effective prompt instructions to guide llms to improve task performance and to better fit the user's needs has been widely explored by practitioners and documented through resources such as prompting handbooks. the research community has also developed interest in automatically crafting prompts through various studies that in your job is to analyze example pairs of source texts with generated summaries and target summaries . you will write a newline separated short list of sentences ( up to 8) as guidelines on generating better summaries that match the desired text length , formatting , grammatical person , level of abstraction and sentence complexity shown in the target examples , while avoiding the mistakes present in the generated summaries . these guidelines will help a summarizer produce better summaries without seeing any of the example summaries . focus on general principles , not specific example details or narrow sub -domain instructions . example { i }: source text : { source_text } generated summary : { zero_shot_summary } target summary : { reference_summary } write a newline separated short list of sentences ( up to 8) as guidelines on generating summaries that better match the desired text length , formatting , grammatical person , level of abstraction and sentence complexity shown in the target examples , while avoiding the mistakes in the generated summaries . keep the list short . only produce the guidelines and no additional text . never mention specific examples , target summaries , or generated summaries in the guidelines . the guidelines should be general and applicable to the dataset as a whole , providing clear direction that can be followed independently . guidelines : prompt 1: prompt used to generate guidelines. we always use two examples. we now introduce some relevant works in autoprompting: @xcite explore alternative phrasings of the initial prompt through word-and phrase-level edits. @xcite iteratively refine the initial prompt using natural language "batch gradients" that critique the current prompt whilst this prompt is being adapted to the opposite semantic direction of the gradient. in parallel work, literature of in-context reinforcement learning has demonstrated that llms can improve and even acquire new abilities by receiving numerical or verbal feedback on their past generations often through multiple selfiterations @xcite @xcite . our approach builds on the auto-prompt tuning paradigm by incorporating insights from in-context reinforcement learning. however, unlike iterative approaches, our method adapts the summarization prompt to a new domain through a single preparatory step that requires only two development samples and two summaries generated through a generic prompt for a total of just three llm inferences. ## results this section presents our results in five parts. we begin with the rouge evaluation of our main results, followed by an llm preference evaluation. subsequently, we measure the agreement between rouge and the llm preference within the subset. afterwards, we analyze a pitfall and an opportunity of our guidelines approach and conclude with remarks on the characteristics/irregularities of our generated text. ## discussion computational efficiency our approach to generating summarization guidelines has similar attributes to in-context learning prompting but with a computational cost that is close to a minimal prompt. we process the two demonstration examples once to produce reusable guidelines, so we do not have to include the k-shot examples at inference time. for @xmath0, that means we can omit prepending the two source texts and their reference summaries every time we perform summarization. this efficiency gain is especially prevalent in long-form summarization. for instance, in multi-lexsum the two source texts plus their reference summaries can add up to about 190,000 input tokens, whereas the resulting guidelines are roughly 300 tokens. when performing summarization at scale, this minimization of input tokens would significantly reduce costs. flexible initialization in this work, we focus on deriving summarization guidelines with an llm by comparing model-generated summaries to reference summaries. that said, the guidelines can also be initialized in other ways, for example, from a combination of editorial standards and generated summaries, or from generated summaries paired with human post-edits. in practice, the guidelines could even be written entirely by humans; at that point, the process is more accurately described as prompt engineering. human post-editable because the llmproduced summarization guidelines are written in natural language, the patterns and guidelines inferred from the provided samples can be reviewed and customized by a subject matter expert. we therefore propose a human-llm collaboration framework comprising three steps: ## conclusions we present a framework that adapts summarization models to new domains using only two reference summaries and three llm inferences. the three evaluation domains: lawsuits, arxiv papers, and patents are deliberately distinct, motivating the need for domain-sensitive adaptation. across these domains, the method yields mostly consistent gains in rouge and llm preference. guideline-based summaries are preferred in the majority of the evaluations across 4b, 12b, and 27b models. the selfreflection step enables models to compare their outputs with references, derive actionable, domainspecific guidelines, and use those guidelines in prompts to consistently outperform zero-shot and standard in-context learning baselines. our joint evaluation shows low agreement between rouge and llm preference (cohen's kappa -0.05 to 0.20), suggesting that structural and coherence improvements favored by llm judges are not captured by token-overlap metrics. minimal examples, when converted into self-generated guidelines, enable effective domain adaptation without costly few-shot setups while maintaining competitive quality. the approach also yields interpretable guidelines and may inform future work on adaptive deployment and the role of self-reflection in domain-specific summarization.