Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals? (2024)

Abstract: In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.

## introduction recent work on the evaluation of large language models (llms) has shown unprecedented performance on diverse language generation tasks, even in zero-shot settings @xcite . specifically in text summarization, @xcite found that human evaluators prefer summaries produced by gpt-3 over strong supervised baselines. in a similar experiment testing ten large language models, @xcite concluded that instruction-tuned models performed on par with human freelance writers. additionally, their results suggest that the evaluation of news summarization is hindered by the low quality of reference summaries. hence, the conventional summarization figure 1 : an overview of our controllability experiments. we expose llm summarizers to prompts conveying communicative intentions related to conciseness, narrative perspective, and keywords inferred by a keyword model. then, we measure how generated summaries adhere to those intentional targets. to address the first question, we design an experiment to compare human and machine summarizers, in which human summaries are judged in terms of their similarity with respect to another set of human-written reference summaries. by evaluating the lexical overlap with reference summaries from the multi-perspective scientific summarization dataset -mup @xcite , we find that human reviewers achieve lower rouge scores @xcite compared to llms. the high quality of llm summaries is confirmed by a human judgment experiment, where machine-generated summaries are preferred in 83% of the instances. the strong preference for llm summaries hints that the variability in communicative intentions @xcite outweighs the usual quality criteria such as coherence, fluency, and relevance @xcite . in fact, an inspection of human summaries for the same article reveals arbitrary decisions related to conciseness and coverage of scientific aspects. we argue that those subjective decisions affect the perceived quality of summaries and that evaluation protocols should consider the adaptability of summarizers to diverse contexts @xcite . motivated by the experiment on the mup dataset and previous literature on controllable summarization @xcite , we elect three intentional aspects: conciseness, narrative perspective (first or third person), and keyword coverage. for each of those aspects, we define two elements: intention prompts, and intention control metrics. intentional prompts are designed to specify the task and to elicit a given intentional behavior from llms (e.g., summarize this article in 5 sentences). finally, the control metrics assess the intentional alignment of prompts and the generated summaries. to investigate the second research question, we evaluate our intentional prompts on the tasks of abstract generation @xcite and lay summarization @xcite , using gpt-3.5 and llama-2 @xcite . we find that both models can follow conciseness and narrative perspective intentions accurately to generate short abstracts for arxiv and pubmed. furthermore, by using keyword-based intentional prompts and classifier-free guidance @xcite , we can direct to the lexical content of summaries leading to rouge scores comparable to strong supervised baselines. however, the elife lay summarization benchmark presents a harder challenge for llms. besides being highly abstractive, these lay summaries are much longer (18 sentences on average), and have a particular concept distribution emphasizing research background. our controllability results show that intentional prompts achieve limited success in replicating these characteristics. overall, our experiments spanning four benchmarks indicate that llms are effective summarizers for long scientific documents, both in terms of lexical alignment and human preferences. however, when summarization tasks deviate from the training distribution (e.g., lay summarization), llms cannot consistently match the features of human summaries. thus, llms do not usher the "death" of summarization @xcite but the transition from reference-based evaluation to more nuanced, domain-specific evaluation protocols, as envisioned by jones (1998) 25 years ago. ## guiding summarizer intentions as uncertainty in intentionality is intrinsic in language generation @xcite , we argue that summarization performance relates to the capacity of a system to adapt its behavior given not only the source documents but also the target communicative intentions. we investigate language model intent adaptability by changing the summarization context via prompting, and measuring how it affects the perceived intention in the summaries (see figure 1 for an example). let p θ be a language model parameterized by θ. then, we define summarization as a conditional sequence generation: in our experiments, we consider that paper abstracts from arxiv and pubmed use firstperson narrative, and other summaries such as paper reviews from mup @xcite and lay summaries from elife @xcite use third person narrative. since we specify intention guidance via prompts, we have no control over how the language models trade off their unconditional summarization behavior and the intention instructions. to address this issue, we modify the decoding process in eq. 2 to include classifier-free guidance (cfg) weighting @xcite : write a summary of the article above. when γ > 1, the next-token probabilities are changed so that the generated summary is closer to the target intentions than its default summarization behavior elicited by the prompt p ϵ . to measure the adherence of summaries to intention prompts, we define reference-free intention control metrics k i (y i , s) that gauge the intentional alignment of s with respect to a target value y i . in the next sections, we detail the intention prompts p i and control metrics k i used in our experiments. ## experiments in this section, we present the experimental settings spanning three styles of scientific summaries: abstracts, reviews, and lay summaries. ## results and discussion in this section, we discuss our experimental results comparing human and llm-generated summaries (section 4.1), and the controllability experiments on the abstract generation and lay summarization tasks (section 4.2). ## related work our work fits in the context of recent research of large language models applied to summarization, mostly for news articles @xcite . in this work, we explore summarization in the scientific domain and its specific model fkgl↓ novel bigram↑ reference 10.8 66.6 bart 10.2 57.8 write a lay summary of the article above. llama 10.9 47.6 "non-intentional" baseline prompt p ϵ llama 15.69 31.5 p conciseness llama 14.8 28.4 llama-cfg 14.1 27.0 p conciseness + p keywords llama 10.9 47.2 llama-cfg 12.8 55.8 previous work has explored summarization controllability, including length control and entity coverage @xcite . in this work, we explore similar summary features and expand our analysis to the challenging setting of lay summarization. this task requires the production of long, highly abstractive summaries that cover more background information. several other challenging scientific summarization tasks (e.g., focusing on paper contributions and related literature) would lead to promising research directions. finally, our work is related to attempts to improve the prompt-adherence in generation tasks. @xcite apply prompt-based methods to control summarization according to target audience (expert vs. layman) and style (formal vs. informal). @xcite propose a nonautoregressive generation method to introduce soft and hard constraints, including adherence to keywords. in a different direction, @xcite showed that context-free guidance (cfg) @xcite , a technique originally used in text-to-image models, improves the performance of language models across several tasks. while @xcite applied cfg to tasks such as question answering, code generation and translation, our work is the first to explore this technique for summarization controllability. ## conclusion our experimental results on four scientific summarization benchmarks confirm that large language models are effective summarizers, as measured by conventional lexical overlap metrics and human preferences. moreover, we find that llms can follow intentional prompts for style and keyword coverage, especially for short summaries. this summarization controllability can be further improved using simple decoding changes such as classifierfree guidance. however, longer summaries that require a higher level of abstractiveness are not easily achievable with our prompting techniques. our findings suggest that analyzing summaries as an expression of communicative intentions leads to informative and actionable insights for future model improvements, where traditional reference-based evaluation shows its limits.