Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023)

Abstract: In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, we introduce Socratic pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks. By training a model to generate and answer relevant questions in a given context, Socratic pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized. We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data. Furthermore, our results show that Socratic pretraining cuts task-specific labeled data requirements in half, is more faithful to user-provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.

## introduction summarization systems are designed to help users navigate large amounts of information @xcite , but often fail to meet the unique needs of different users, especially for long documents. recent research has explored ways to make summarization systems more controllable @xcite by allowing users to input queries or control sequences such as keywords @xcite , questions @xcite , entity chains @xcite , or question-answer pairs @xcite . a challenge shared by all of the mentioned approaches is the absence of abundant labeled data. ## socratic pretraining during socratic pretraining, the model takes as input a document with important sentences masked and is trained to generate questions about the masked content and produce the mask itself. as seen in figure 1 , socratic pretraining is formulated as a sequence-to-sequence task and consists of two steps 1) important content is selected from unlabeled documents to be masked, and 2) a question-generation system is applied to produce questions about the selected content. the question augmentation component trains the model to produce summaries grounded to questions and allows for controllability as the end-user can prompt the model decoder with new questions during inference. we describe both steps below. ## experimental setup we describe the experimental setup that we use to study socratic pretraining along with empirical studies justifying our design decisions. ## socratic pretraining ablations in this section, we corroborate our design choices with ablation studies of the components of so-cratic pretraining. similar to @xcite and @xcite , to save time and resources, we conduct the ablations of the objective on a small scale by restricting the pretraining adaptation to 1m documents from the openwebtext2 corpus and then finetuning it on the full downstream task datasets. we report the mean over five randomly initialized finetuning runs on the validation set. ## query focused summarization results we scale the socratic pretraining adaptation based on the findings of the previous ablation and evaluate its downstream effects on query-focused summarization. unless specified, the results in this section are averaged over five randomly initialized finetuning runs on the downstream tasks. in ## conclusion in this work, we introduce socratic pretraining, a question-driven, unsupervised pretraining objective to adapt generic language models to the task of controllable summarization. socratic pretraining trains the model to generate relevant questions in a given context and then to answer them. our experiments demonstrate the generality of our approach both on query-focused summarization and finegrained controllable summarization. we show that socratic pretraining outperforms other pretraining and prefinetuning objectives, that it cuts downstream task data requirements in half, and that it works across control strategies and domains.