D 2 S : Document-to-Slide Generation Via Query-Based Text Summarization (2021)

Abstract: Presentations are critical for communication in all areas of our lives, yet the creation of slide decks is often tedious and time-consuming. There has been limited research aiming to automate the document-to-slides generation process and all face a critical challenge: no publicly available dataset for training and benchmarking. In this work, we first contribute a new dataset, SciDuet, consisting of pairs of papers and their corresponding slides decks from recent years’ NLP and ML conferences (e.g., ACL). Secondly, we present D2S, a novel system that tackles the document-to-slides task with a two-step approach: 1) Use slide titles to retrieve relevant and engaging text, figures, and tables; 2) Summarize the retrieved context into bullet points with long-form question answering. Our evaluation suggests that long-form QA outperforms state-of-the-art summarization baselines on both automated ROUGE metrics and qualitative human evaluation.

## introduction from business to education to research, presentations are everywhere as they are visually effective in summarizing and explaining bodies of work to the audience @xcite @xcite . however, it is tedious and time-consuming to manually create presentation slides @xcite . researchers have proposed various methods to automatically generate presentations from source documents. for example, @xcite suggest heuristic rule-based mechanisms to extract document contents and use those as the generated-slide's content. ppsgen @xcite to learn a sentence's importance in the document, and extract important sentences as slide's content. these existing research works have yielded promising progress towards the goal of automated slide generation, but they also face two common limitations: 1) these works primarily rely on extractive-based mechanisms, thus the generated content is merely an aggregation of raw sentences from the document, whereas in real-world slides, the presenter frequently uses abstractive summarization; 2) these works assume the presentation slide's title has a one-to-one match to the document's subtitles or section headlines, whereas the presenter in reality often uses new slide titles and creates multiple slides under the same title (e.g., the slides with a cont. / continue on it). in our work, we aim to tackle both the limitations. to achieve this goal, we consider the document-to-slides generation task as a query-based single-document text summarization (qss) task. our approach leverages recent research developments from open-domain long-form question answering (qa). specifically, we propose an interactive two-step architecture: in the first step, we allow users to input a short text as the slide title and use a dense vector ir module to identify the most relevant sections/sentences as well as figures/tables from the corresponding paper. then, in the second step, we use a qa model to generate the abstractive summary (answer) of the retrieved text based on the given slide title and use this as the final slide text content. we design a keyword module to extract a hierarchical discourse structure from the paired paper. for a given title, we leverage leaf nodes from this tree structure in our ir module to rank paper snippets. we further extract related keywords from this structure and integrate them into the qa module. experiments demonstrate that the keyword module helps our system to retrieve more relevant context and generate better slide content. it is worth noting that our system can extract relevant figures and tables for a given title from the source document as well. figure 1 (bottom) shows an example of a generated slide from our system. in addition to our contribution of the novel model architecture, we also contribute a high-quality dataset (sciduet), which contains 1,088 papers and 10,034 slides. we carefully build this dataset by leveraging a few toolkits for pdf parsing and image/table extraction. to the best of our knowledge, this is the first publicly available dataset for the document-to-slides generation task foot_0 . our dataset together with the title-based document-toslide generation task provide a practical testbed for the research field on query-based single-document summarization. we release the dataset procurement and preprocessing code as well as a portion of sciduet 2 at https://github.com/ibm/ document2slides. ## related work automated document-to-slides generation the early works of automatically generating presentation slides date back to 20 years ago and rely on heuristic rule-based approaches to process information from web searches as slide contents for a user-entered topic @xcite . a recent example, winters and mathewson (2019) used predefined schemas, web sources, and rule-based heuristics to generate random decks based on a single topic. among this group of works, different types of rules were used, but they all relied heavily on handcrafted features or heuristics @xcite @xcite . more recently, researchers started to leverage machine learning approaches to learn the importance of sentences and key phrases. these systems generally consist of a method to rank sentence importance: regression @xcite @xcite , random forest @xcite , and deep neural networks @xcite . and they incorporate another method for sentence selection: integer linear programming @xcite @xcite and greedy methods @xcite . however, these methods all rely on extractive approaches, which extract raw sentences and phrases from the document as the generated slide content. an abstractive approach based on diverse titles that can summarize document content and generate new phrases and sentences is under-investigated. ## sciduet dataset construction data sources the sciduet (scientific document slide match) dataset comprises of paperslide pairs scraped from online anthologies of international conference on machine learning (icml'19), neural information processing systems (neurips'18&'19), and association for computational linguistics (since acl'79) conferences. we focus only on machine learning conferences as their papers have highly specialized vocabulary; we want to test the limits of language generation models on this challenging task. nevertheless, these generic procuration methods (web-scraping) can be applied to other domains with structured archives. data processing text on papers was extracted through grobid @xcite . figures and #papers #slides st-len sc-len train 952 8,123 3.6 55.1 dev 55 733 3.16 63.4 test 81 1,178 3.4 52.3 text on slides was extracted through ibm watson discovery package foot_2 and ocr by pytesseract. foot_3 figures and tables that appear on slides and papers were linked through multiscale template matching by opencv. further dataset cleaning was performed with standard string-based heuristics on sentence building, equation and floating caption removal, and duplicate line deletion. dataset statistics and analysis sciduet has 952-55-81 paper-slide pairs in the train-dev-test split. we publicly release sciduet-acl which is constructed from acl anthology. it contains the full dev and test sets, and a portion of the train dataset. note that although we cannot release the whole training dataset due to copyright issues, researchers can still use our released data procurement code to generate the training dataset from the online icml/neurips anthologies. introduction related work …… experiments …… h1 h2 h3 h4 h5 h6 segment 1 segment 2 segment 3 …… qa module query context experiments segment 1 segment 3 …… h3 h4 h5 h6 ## d2s framework we consider document-to-slide generation as a closed-domain long-form question answering problem. closed domain means the supporting context is limited to the paired paper. while traditional open-domain qa has specific questions, nearly 40% of our slide titles are generic (e.g., take home message, results). to generate meaningful slide contents (answers) for these titles (generic questions), we use title-like keywords to guide the system to retrieve and generate key bullet points for both generic titles and the specific keywords. the system framework is illustrated in figure 2 . below, we describe each module in detail. ## experimental setup implementation details all training was done on two 16gb p100 gpus in parallel on pytorch. our code adapts the transformer models from hug-gingface @xcite . all hyperparameters are fine-tuned on the dev set. a distilled uncased bert miniature with 8-layers, 768 hidden units, and 12 attention heads was trained and used to perform ir. the bert model computes all sentence embeddings in 128-dimensional vectors. our qa model was fine-tuned over bart-large-cnn, a bart model pre-trained on the cnn-dailymail dataset. pilot experiments showed that bart-large-cnn outperforms bart-large and other state-of-the-art pre-trained language generation models on the dev dataset. the bart model used the adamw optimizer and a linear decreasing learning rate scheduler. during testing, we apply our trained qa model with the lowest dev loss to the testing dataset. note that we do not apply any content filtering on the testing data. the qa models generate the predicted slide content using beam search with beam 8 and no repeated trigrams. we use the dev dataset to tune the minimum and maximum token lengths for the output of the qa model. evaluation we evaluate our ir model using idfrecall, which computes the proportion of words in the original slide text in the retrieved context weighted by their inverse document frequency. this metric gives more focuses to important words. for adaptive figure selection, we report the top-(1, 3, 5) precision. finally, for slide text content generation, we use rouge as the automatic evaluation metric @xcite . we also carried out human evaluation to assess the d2s system's performance on slide text generation. ## evaluation on ir and figure selection results on dense ir for a given slide title, the goal of ir is to identify relevant information from the paired paper for the downstream generation model. we compare our ir model (dense-mix ir) described in section 4.2 to a few baselines. classical ir (bm25) is based on sparse word matching which uses the bm25 similarity function. dense-text ir and dense-keyword ir are variants of our dense-mix ir model with different ranking functions (α equals 1 for dense-text ir and 0 for dense-keyword ir). all experiments are evaluated on the test set. the idf-recall scores for each ir method are as follows: classical ir (bm25) = 0.5112, dense-text @xmath0.5476, dense-keyword @xmath1.5175, and dense-mix @xmath2.5556. the experiments indicate the dense ir model outperforming the classical ir approach and an α = 0.75-weighted mix dense ir model outperforming other dense ir models that rank exclusively by text or keywords. these results support the design decision of using embedding-based ir and re-ranking based on both text snippets and keywords. we attribute the success of the dense-mix ir model to increased section header awareness. header-awareness leads to better retrieval in cases where the title corresponds well with section headers. the drawback of ranking solely on keywords is in the case when the dense ir module cannot differentiate between passages with the same header. this leads us to find the right balance (α = 0.75) between dense-text ir and dense-keyword ir. results on figure selection we evaluate figure selection based on the set of the testing slides which contain figures/tables from the paired papers. the results of the adaptive figure selection are promising. it achieves 0.38, 0.60, and 0.77 on p@1, p@3, and p@5, respectively. this suggests our system is holistic and capable of displaying figures and tables for slides that the original author chose. ## system analysis in this section, we carry out additional experiments to better understand the effectiveness of different components in our system. ## error analysis to gain additional insights into our model's performance, we carried out a qualitative error analysis to check the common errors in our best system (dense-mix ir + bartkeyword). we sampled 20 slides that received lower rating scores (rating @xmath3.2.2). one author of this paper carefully checked each generated slide content and compared it to the original paper/slide. in general, we found that most errors are due to off-topic content. for instance, given a slide title "future work", our model might generate sentences that summarize the major contributions of the corresponding paper but do not discuss next steps. we also observed that occasionally our model hallucinates content which is not supported by the corresponding paper. normally, this happens after the model selects an example sentence from the paper and the sentence's content is very different from its surrounding context. for instance, a paper uses an example sentence "which cinemas screen star wars tonight?" to illustrate a new approach to capture intents/slots in conversations. then for the slide title "reminder q&a data", our model generates "which cinemas screen star wars tonight? which movie theater plays star wars at 8 p.m. on december 18?". here, the second sentence is a hallucination error. we use the novel n-grams to measure the "abstractiveness" of the generated slide contents. on the testing dataset, we found that the original slide contents contain a much higher proportion of novel n-grams compared to the automatically generated ones (e.g., 24.2% vs. 3.1% for novel unigrams, and 66.5% vs. 14.2% for novel bigrams). this indicates that the generated slide contents from our model are still mostly "extractive". ## conclusion this project aims to automatically generate presentation slides from paper documents. the problem is framed as a query-based single-document summarization task. inspired by recent work on opendomain long-form qa, we design a keyword-aware framework (d2s) to tackle this challenge. both automated and human evaluations suggest that our system outperforms a few strong baselines and can be served as a benchmark for the document-to-slide challenge. we release the dataset (sciduet) and code in hopes it can foster future work. sida wang, xiaojun wan, and shikang du. 2017. phrase-based presentation slides generation for academic papers. in proceedings of the thirty-first aaai conference on artificial intelligence, pages 196-202. yuanyuan wang and kazutoshi sumiya. 2013. a method for generating presentation slides based on expression styles using document structure. international journal of knowledge and web intelligence, 4(1):93-112. thomas winters and kory w mathewson. 2019. automatically generating engaging presentation slide decks. in international conference on computational intelligence in music, sound, art and design (part of evostar), pages 127-141. springer. thomas wolf, julien chaumond, lysandre debut, victor sanh, clement delangue, anthony moi, pierric cistac, morgan funtowicz, joe davison, sam shleifer, et al. 2020. transformers: state-of-theart natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45. yumo xu and mirella lapata. 2020. coarse-to-fine query focused multi-document summarization. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 3632-3645, online. association for computational linguistics.