Can Language Models Capture Human Writing Preferences for Domain-Specific Text Summarization? (2025)

Abstract: With the popularity of large language models and their high-quality text generation capabilities, researchers are using them as auxiliary tools for text summary writing. Although summaries generated by these large language models are smooth and capture key information sufficiently, the quality of their output depends on the prompt, and the generated text is somewhat procedural to a certain extent. We construct LecSumm to verify whether language models truly capture human writing preferences, in which we recruit 200 college students to write summaries for lecture notes on ten different machine-learning topics and analyze writing preferences in real-world human summaries through the dimensions of length, content depth, tone & style, and summary format. We define the method of capturing human writing preferences by language models as finetuning pre-trained models with data and designing prompts to optimize the output of large language models. The results of translating the analyzed human writing preferences into prompts and conducting experiments show that both models still fail to capture human writing preferences effectively. Our LecSumm dataset brings new challenges to finetuned and prompt-based large language models on the task of human-centered text summarization.

## introduction with the vast amount of training data, the development of large language models (llms), such as the gpt series @xcite , the palm series (aakanksha chowdhery, 2022), mistral @xcite and llama (ai@meta, 2024), have achieved remarkable success by unifying the generative paradigm with different nlp tasks @xcite @xcite @xcite . in specific nlp fields, such as text summarization, llms achieve decent performance without additional training data and even surpass traditional supervised finetuned models @xcite . recent studies employ llms as auxiliary tools for humancentered nlp @xcite , ranging from human-centered design to human-inthe-loop interaction with llms. when generating human-centered summaries with llms, specific human preferences can be incorporated through two different approaches : (i) explicitly, add external constraints to the summarization model, such as prompt design and different hyperparameter settings. (ii) implicitly, construct specific sourcetarget summary datasets that reflect human preferences to finetune the language model, enabling it to learn the hidden preferences from the data. our research question is: can language models really capture human writing preferences through prompting and in-context training? to answer this question, we first design a lecture note summarization task to discover human preferences in real-world data and construct a dataset containing human-centered summaries. the task framework is shown in figure 1 . we recruited 200 university students and designed a task of writing lecture note summaries: 10 different topics related to machine learning were given to the annotators, together with the corresponding lecture notes, and the participants were required to write summaries based on the lecture notes. there is no hard limitation (e.g., length, summary format) on the summary writing process; participants are allowed to use related materials to equip the lecture notes. the only limitation is that it has to be written by the participants and cannot be returned by machines. we observe that different annotators utilize a combination of various dimensions to reflect their writing preferences; these human preferences range in four dimensions: length, content, tongue&style, and summary format. then, we construct the lecsumm dataset, which includes the provided lecture notes and humanwritten summaries, and experiment with finetuned supervised models and prompt-based zero-shot llms. we find that language models can not capture human writing preferences well through model finetuned and prompt design. our main contribution is presented as follows: