Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning (2024)

Abstract: The evaluation of summary quality encompasses diverse dimensions such as consistency, coherence, relevance, and fluency. However, existing summarization methods often target a specific dimension, facing challenges in generating well-balanced summaries across multiple dimensions. In this paper, we propose multi-objective reinforcement learning tailored to generate balanced summaries across all four dimensions. We introduce two multi-dimensional optimization (MDO) strategies for adaptive learning: 1) MDO_min, rewarding the current lowest dimension score, and 2) MDO_pro, optimizing multiple dimensions similar to multi-task learning, resolves conflicting gradients across dimensions through gradient projection. Unlike prior ROUGE-based rewards relying on reference summaries, we use a QA-based reward model that aligns with human preferences. Further, we discover the capability to regulate the length of summaries by adjusting the discount factor, seeking the generation of concise yet informative summaries that encapsulate crucial points. Our approach achieved substantial performance gains compared to baseline models on representative summarization datasets, particularly in the overlooked dimensions.

## introduction determining a "good summary" extends beyond a single factor, generally embracing multiple dimensions such as coherence, consistency, fluency, and relevance @xcite @xcite @xcite . despite the remarkable advancements in abstractive summarization, challenges persist in addressing issues such as factual inconsistency, which generates inaccurate information, and irrelevance, which involves omitting crucial details. ## method description throughout the rl process, it is crucial to maintain the fundamental summarization capabilities of the fine-tuned model while simultaneously improving scores across various dimensions. to achieve this goal, we employ proximal policy optimization (ppo) @xcite for rl application, utilizing a supervised, parameter-frozen reference model to guide the policy. in our pursuit of multiobjective rl in summarization, we adopt unieval @xcite , a metric that evaluates scores across different dimensions using a qa model. incorporating four dimensions in the rewarding process, we introduce two optimal mdo methods to guide rl policy updates effectively. the entire process is illustrated in figure 2 . ## experimental setup datasets we utilize two text summarization datasets considering potential influences of source document complexity: the billsum dataset for legislative content and the cnn/daily mail dataset for news summarization. billsum comprises an 18.9k training set and a 3.2k test set, while cnn/dm has a 287k training set and an 11.5k test set. in light of studies indicating poor quality of reference summaries in the datasets @xcite @xcite , we use an enhanced version of cnn/dm test set introduced by @xcite . baseline models as baseline models, we employ encoder-decoder models commonly used for the text summarization task, including bart @xcite and t5 @xcite . for additional comparison, we report pegasus @xcite and brio @xcite results. to ensure comparability, we fine-tune bart-base, bart-large, and t5-base under the same hyperparameter settings: a batch size of 4, a learning rate of 5e-5, and 10 epochs. for pegasus and brio models, we utilized already fine-tuned versions on the billsum foot_0 and cnn/dm 23 . in addition, we compare our model with llms, gpt-3-cot @xcite and @xcite . gpt-3-cot is a 2-stage chain-ofthought approach where the first stage extracts the core elements, and the second stage integrates them to address the issue of llms not sufficiently incorporating elements in generated summaries in the news datasets. we used gpt-4-turbo for gpt-4. hyperparameters for rl for rl, we use a batch size of 4, a learning rate of 1.41e-6, discount factor 𝛾 = 0.9, and randomly select only 10k samples from the training set of each dataset. we conduct experiments with three different seeds and report the average scores. ## discussions summary length varies by text complexity in text summarization tasks, concisely encapsulating only the critical information is crucial. however, the optimal length of a summary depends on the document's informational content, resulting in varying ideal lengths across datasets. when a document contains rich information, its summary tends to be longer; conversely, a document with less information leads to a shorter summary. the cnn/dm news dataset includes less information, allowing for the essential contents to be sufficiently covered in a shorter length. on the other hand, the legislative dataset, billsum, characterized by longer texts and a greater volume of information, tends to yield longer summaries for all models, as revealed in figure 6 . remarkably, our models consistently produce short yet concise summaries for both datasets, while the pegasus model outputs severely lengthy summaries when the data complexity increases.