C oco S ci S um: A Scientific Summarization Toolkit with Compositional Controllability (2023)

Abstract: We present a novel toolkit for controlled summarization of scientific documents, designed for the specific needs of the scientific community. Our system generates summaries based on user preferences, adjusting key attributes specifically of length and keyword inclusion. A distinguishing feature is its ability to manage multiple attributes concurrently, demonstrating Compositional Controllability for Scientific Summarization (CocoSciSum). Benchmarked against the strong Flan-T5 baseline, CocoSciSum exhibits superior performance on both the quality of summaries generated and the control over single and multiple attributes. Moreover, CocoSciSum is a user-centric toolkit, supporting user preferences expressed in natural language instructions, and accommodating diverse input document formats. CocoSciSum is available on GitHub (https://github.com/WING-NUS/SciAssist/tree/CocoSciSum) with an introduction video (https://youtu.be/YC1YDeEjAbQ).

## introduction scientific summarization refers to the process of distilling scientific documents such as research papers into shorter versions that capture the key information. it has become increasingly important as it can facilitate quick search results filtering, as seen in semantic scholar's tl;dr (too long, didn't read) feature. however, we argue that controlled summarization for scientific documents, which generates user-customized summaries over various control attributes, can further enhance personalized result filtering and paper comprehension. for example, a reader primarily interested in a paper's relationship to a particular term -say, 'cloze task' -can instruct the summarization system to focus on the term when generating the summary. accordingly, we introduce the first toolkit designed for controlled summarization in the scientific domain. it consistently generates high-quality summaries, as validated by various automatic evaluation metrics and human annotators. aside from improved performance, our system, cocoscisum, makes two key contributions to the summarization landscape. contribution 1: compositional controllability. we seek a unified solution for all summarization functions, aiming to reduce memory usage and streamline deployment. cocoscisum provides three modes of summarization: (1) vanilla summarization, (2) single-attribute controlled summarization (managing either length or keyword inclusion), and (3) compositionally-controlled summarization, which jointly regulates both length and keywords. it employs instruction-tuned plms; specifically adopting @xcite as the backbone model. figure 1 shows the framework of the summarization model. to achieve such compositional goals, past summarization systems all have resorted to separate models to control each attribute of interest @xcite @xcite @xcite . however, this presents a significant challenge to the pre-trained language model (plm) finetuning paradigm, due to the scarcity of training data. the necessary creation of training data incurs significant annotation costs, given the complexity of annotating multiple reference summaries for a single document, each conforming to individual or multiple attribute constraints. the disparity between available and desired training data compels us to seek an alternative, as reference summaries for multiple controls are not available and prohibitively expensive to annotate. instead, we break down the annotation task into simpler sub-tasks, synthesizing summaries constrained for each attribute separately. we hypothesize that by initially training the model on simplified relevant tasks, it can subsequently generalize. that is, we finetune the plm using single-attribute constrained data, expecting its capability to solve composite summarization tasks. contribution 2: user-friendly. we build co-coscisum to allow easy uptake for those simply curious as well as serious practitioners. we offer a demonstration page 3 for cocoscisum's immediate utilization, and an easy-to-install python package along with comprehensive development documentation 4 for practitioners. cocoscisum itself is also user-friendly: it interprets user preferences specified in natural language instructions and supports multiple input document formats. in contrast to previous work that employed opaque vectors (i.e., unreadable to humans) for summarization control, cocoscisum adopts user-friendly natural language to specify instructions, such as "give a summary within 50 words and containing the phrase 'automated evaluation technique': " (figure 1 ). finally, considerating the prevalence of portable document format (pdf) for scientific documents (which are not directly machine-readable), we incorporate a specialized pdf text extraction module. our toolkit also supports other document formats like plain text and structured json for wider practical usage. ## architecture we show the architecture of cocoscisum in figure 2. it consists of training and inference phases. configuration. hydra @xcite is an opensource python framework that simplifies the management of configurations in applications. in particular, hydra allows dynamically creating a hierarchical configuration by composition from multiple sources, and overriding it through both the original configuration files and the command line, facilitating specifying hyper-parameters and experimental settings. we follow a deep learning project template foot_1 for designing hydra in cocoscisum. utility. the utility module centralizes all data processing procedures, inclusive of tokenizing text-label mixed strings (tokenize_and_align_labels), generating the text sequence from a matrix of probabilities over a vocabulary (prob2text) and customizing batch collation (batch_collator). all data utility functions are encapsulated into a datautils class, step 3 step 1a: upload your .pdf or .txt file. step 1b: or simply paste your text string. you can check the extracted text from your file here if your input is not a string. which facilitates the use of different datasets in various formats, and enables the seamless application of a single dataset to multiple models. ## compositionally controlled summarization we now introduce the compositionally-controlled summarization model used in cocoscisum. the model is designed to support all summarization functions, including standard, single-attributecontrolled and multi-attribute-controlled summarization. we first introduce the backbone summarization model and then detail the instruction tuning for compositional controllability. ## conclusion we present the first scientific toolkit, cocoscisum, which summarizes scientific papers by customizing multiple aspects, including fine-grained length control, keyword inclusion, and compositional control of both. the user preferences are specified in natural language instructions rather than humanunreadable vectors. we provide an easy-to-install python package and a demo page for different uses. cocoscisum supports multiple input formats of scientific papers, plain text, structured json, and the most commonly used pdf. we employ flan-t5 as the backbone model, finetuning it on singleattribute-controlled tasks, and prove its capability for multi-attribute control. in future work, we will explore compositional controllability conflicts and user-specified weighting between multiple attributes. signals change. for evaluating keyword controllability, we use (1) the success rate (sr, @xcite 9 to reflect the fraction of requested keywords actually occurring in the output summaries; and (2) the factual correctness (fact, @xcite to ensure the generated summaries contain factually accurate information. we introduce the calculation of the evaluation metrics as follows. mean of absolute deviation (mad, @xcite is used to evaluate the distance between the target length l target and the length of the systemgenerated summary l sys . we categorize summaries into bins based on their lengths: bin 1 contains 0-50 words, bin 2 has 50-100 words, and so forth. we calculate mad using the bin number of corresponding length with the following equation: pearson correlation coefficient (pcc, @xcite , which is a number between -1 and 1 that measures the strength and direction of the relationship between two variables. a number between 0 and 1 means when one variable changes, the other variable changes in the same direction, and vice versa. for each test instance, we generate five summaries with length signals of 50, 100, 150, 200, and 250 words. subsequently, we compute the pcc between the actual length and the control signal for each summary. success rate (sr, @xcite , is the fraction of keywords actually occurring in the output summaries. we calculate sr employing exact matching after stemming. factual correctness (fact, @xcite , is a fine-grained evaluation by human annotators. we ask annotators to make a binary 0/1 judgment for the correctness of each sentence, based on whether it can be entailed or logically inferred from the provided source document. we use voting to decide the final score of the sentence based on judgments of multiple annotators. the fact score of a batch of generated summaries is the average number of binary scores of all sentences. we recruit 3 students from our university for annotation, who are majored in computer science and fluent in scientific document reading in english. these annotators are carefully selected based on 9 full article success rate in @xcite their expertise and demonstrated high performance in a trial task, ensuring the quality and reliability of their evaluations. the average annotation time for the three annotators is 18 hours, and we provide reasonable compensation based on their working hours.