Personalized Abstractive Summarization by Tri-agent Generation Pipeline (2024)

Abstract: Tailoring outputs from large language models, like ChatGPT, to implicit user preferences remains a challenge despite their impressive generative capabilities. In this paper, we propose a tri-agent generation pipeline comprising a generator, an instructor, and an editor to enhance output personalization. The generator produces an initial output, the instructor automatically generates editing instructions based on user preferences, and the editor refines the output to align with those preferences. The inference-only large language model (ChatGPT) serves as both the generator and editor, with a smaller model acting as the instructor to guide output generation. We train the instructor using editor-steered reinforcement learning, leveraging feedback from a large-scale editor model to optimize instruction generation. Experimental results on two abstractive summarization datasets demonstrate the effectiveness of our approach in generating outputs that better meet user expectations.

## introduction large language models, exemplified by prominent models such as instructgpt @xcite and chatgpt 2 , have emerged as essential resources in the field of natural language processing (nlp). these models have shown an extraordinary level of proficiency across a broad spectrum of nlp tasks, including machine translation, question answering, and text summarization. in light of their potential to drive further innovation in language-based technologies, the research community has exhibited growing enthusiasm for exploring and advancing large language models. however, despite the impressive generation quality achieved by these models, a persistent challenge lies in tailoring their outputs to meet user's preference @xcite . in several scenarios, it has been observed that the outputs of language models do not consistently satisfy users' preferences or expectations @xcite . a prevalent approach to addressing this limitation involves the careful crafting of prompts to steer the models in producing outputs that better align with users' objectives. nonetheless, as noted in existing research @xcite ), the conventional one-time left-to-right generation process of language models contrasts with the iterative refinement and editing approach commonly employed by humans. furthermore, prior works @xcite have demonstrated the efficacy of the generate-and-edit process compared to one-time generation, even with a single editing iteration. motivated by these findings, this paper explores the integration of large language models (chatgpt) into an automatic iterative editing pipeline. who doesn't love a hot cross bun at easter? in britain we so enjoy the spiced, sweet rolls with a cross on top, that they used to be eaten all year round. that was until queen elizabeth i tried to ban them, possibly finding the cross a little too catholic. the move was greeted with uproar, so she compromised, saying they could only be sold on good friday, at christmas and for burials -and they have been prime easter fare ever since. nowadays supermarkets bring out ever more permutations to tempt customers, from miniature sizes to new flavours like toffee fudge chunk and apple and cinnamon. here is best of the batch this easter...... in contrast to the approach taken by @xcite , where the generation process is decomposed into a generator and a corrector, our methodology involves a three-component decomposition consisting of a generator, instructor, and editor (refer to figure 1 ). this structure allows us to leverage inference-only large models for the complex tasks of content generation and correction, while utilizing smaller models for the simpler task of generating user-specific editing instructions. the instructor is designed to provide targeted directives for editing and refining the initial outputs of the generator. it is initialized by training on human-authored, or oracle, instructions, which can be obtained by the history of user's behaviour. following this, the model is then fine-tuned through editor-steered reinforcement learning, wherein the reward function directly quantifies the degree to which the edited output by the editor align with user preferences, which enhances the model's compatibility with the editor. we choose text summarization as the focal task for evaluating this novel framework, which is to generate concise and informative summary for the given document(s). in this paper, we conduct experimental evaluations on two summarization datasets (defacto @xcite and cnndm @xcite ), focusing on user preference related to factual consistency and coverage. we employ chatgpt as the generator and the editor model. our experiments indicate that with the instructions generated by the small instructor model, the edited output is better aligned with user's preference on both datasets. further experiments on the iterative editing shows that the output can better meet user's needs with more iterations of editing. ## overall pipeline in an effort to enhance the flexibility of the generation pipeline and optimize its compatibility with powerful large language models, we propose a novel decomposition of the generation process into three distinct components, as illustrated in figure 2 . these components include: (1) a generator, responsible for producing the initial output; (2) an instructor, tasked with generating natural language instructions that guide the editing of the initial output toward the direction of user preference; and since it has been demonstrated that large language models can act as both a generator and an editor model, we have chosen to utilize an inferenceonly large language model, specifically chatgpt, as our generator and editor. while it is possible to further fine-tune these large language models to serve as instructors, practical limitations such as computational resources @xcite and access restrictions @xcite may prevent direct fine-tuning, as has been done in previous works @xcite . therefore we propose to train a smaller model with editor-steered reinforcement learning to function as a user-specific instructor (as introduced in section 3), which guides the editor in revising the initial output to achieve better alignment with human expectations. ## editor-steered instructor as introduced above, the central objective of the proposed instructor is to produce precise and actionable instructions that can guide a large language model in correcting the original summary to align more closely with the user's preference. to achieve this, we employ a two-phase training process that is designed to enable the instructor to work synergistically with large language models. specifically, given the document d, an initial summary, denoted as s init , is generated using a generator (either a summarization model or a large language model). the objective of the instructor is to take d and s init as inputs and generate a set of instructions @xmath0, i 2 , ..., i k }, aiming to guide the editor model in generating an edited summary that is more closely aligned with the user's preference. finally, the editor takes d, s init , and i as input and generates a revised summary s edit according to the given instructions. ## experiments we conduct experiments on two distinct datasets, each capturing different facets of user preferences. ## conclusion and future work in this paper, we introduce a novel generation paradigm that decomposes the generation process into three distinct components: the generator, the instructor, and the editor. our approach is specifically designed to harness the capabilities of large language models, while accounting for constraints such as limited access and computational resources, and to facilitate the customization of generated content to align with user preference. through a series of pilot experiments on the task of text summarization, we find that large language models, exemplified by chatgpt, can effectively serve as editors, achieving performance levels comparable to supervised editing models when provided with human-written instructions. nevertheless, it is still challenging for the large language models to generate instructions that are well-aligned with human-authored instructions. to address this challenge, we employ a smaller model as the instructor, which is trained with editorsteered reinforcement learning (rl) with rewards based on the quality of the edited summaries. our experimental results demonstrate the efficacy of this approach in guiding the editor (chatgpt) to produce summaries that are more closely aligned with user expectations. looking ahead, future work will involve extending our experiments to other tasks, such as wiki-editing @xcite , newsediting @xcite , and mathematical problem synthesis @xcite . additionally, we may generate more instruction data using the self-instruct technique @xcite to train a better instructor.