Knowledge-Centric Templatic Views of Documents (2024)

Abstract: Authors seeking to communicate with broader audiences often share their ideas in various document formats, such as slide decks, newsletters, reports, and posters. Prior work on document generation has generally tackled the creation of each separate format to be a different task, leading to fragmented learning processes, redundancy in models and methods, and disjointed evaluation. We consider each of these documents as templatic views of the same underlying knowledge/content, and we aim to unify the generation and evaluation of these templatic views. We begin by showing that current LLMs are capable of generating various document formats with little to no supervision. Further, a simple augmentation involving a structured intermediate representation can improve performance, especially for smaller models. We then introduce a novel unified evaluation framework that can be adapted to measuring the quality of document generators for heterogeneous downstream applications. This evaluation is adaptable to a range of user defined criteria and application scenarios, obviating the need for task specific evaluation metrics. Finally, we conduct a human evaluation, which shows that people prefer 82% of the documents generated with our method, while correlating more highly with our unified evaluation framework than prior metrics in the literature.

## introduction sharing information is vital for communication and discourse across domains, as it allows for knowledge to be disseminated to a wider audience. this is often done by users through documents in multiple formats that nevertheless share some underlying knowledge. a product manager may need to create a requirements spec, a product pitch deck, and an announcement newsletter for the same project. likewise, a person on the job market may create a resume, a cover letter, and a personal website. we consider these documents to be templatic views of the same underlying knowledge. this is equally true for the scientific domain, in which researchers create documents in multiple formats to effectively communicate and showcase their work, -such as through academic papers, conference talks, social media posts, poster presentations, and non-technical blog posts. sharing knowledge in multiple formats broadens the audience and can help bridge the information gap between domain experts, researchers in adjacent fields, and even the general public, leading to greater understanding, collaborations and accelerated progress @xcite . past work on document generation has focused on developing generation and evaluation methods specific to a single document type @xcite @xcite . narrow, custom methods tailored to individual document types are, nevertheless, time consuming to engineer and manage over the long term. for example, in an enterprise setting, it's common to have dozens of occupation-and task-specific documents, each with their own template. additionally, specific trained methods require data that may be expensive to acquire, or even be unavailable entirely. meanwhile, llms have recently shown great success in long document generation @xcite , indicating that this fragmentation of methods may no longer be necessary. thus, our goal is to unify methods for both generating and evaluating templatic views of documents, allowing system designers and engineers to manage and adapt to a range of document types and domains easily and efficiently. we begin by showing that llms are capable of diverse, structured document generation, requiring very little instructional guidance to do so effectively. additionally, a few minor augmentations llm llm 0.19 0.62 0.15 template adaptable evaluation structured intermediate representation input document blog post slides poster user defined structure (style params) figure 1 : visualization of our method to unify the generation and evaluation of templatic views of documents. given an input document, we prompt the llm to generate an intermediate representation. we can use the representation to prompt the model to generate a templatic view of the input document. we then evaluate the generations using our unified evaluation framework. the llm represented in the figure is the same model. to the prompt -such as a structured, intermediate representation, and simple stylistic descriptionscan further improve downstream performance, especially for smaller, less resource intensive models. these findings have important implications on the deployment and scaling of unified, real-world aiassisted document authoring systems. in similar vein, we then introduce template adaptable evaluation (tae), departing from prior work's task specific evaluation methods @xcite @xcite . tae is a unified precision-recall style framework for automatic evaluation that is highly customizable, allowing users to easily integrate existing text-based metrics from the literature into its formulation and tailor it to their specific use case.additionally, this framework allows developers to compare performance across document types, without needing to develop an evaluation metric for each individual template. we evaluate our unified approach for templatic view generation and evaluation on 3 types of documents: slides, posters, and blog posts @xcite @xcite . our experiments demonstrate that using a structured intermediate representation leads to improvements in performance across tasks, with greater gains for smaller language language models. in our human evaluation to validate both our unified document generation method and evaluation metric, we show that annotators prefer the output yielded by the structure-aware generation process 82% of the time and that our evaluation metric correlates more highly with human preference than other popular metrics. a visualization of our methods can be found in figure 1 . we release our code foot_0 to support future research. ## related work there are several areas of related research in nlp that are relevant to the problems of document transformation and evaluation. document summarization has been explored in a number of domains, including news @xcite , literature @xcite , law @xcite , and dialogue @xcite . in the scientific domain, summarization of scientific papers has taken the form of long form summaries @xcite , abstract generation @xcite , conference talks @xcite , and query based summaries @xcite . these summaries can be either extractive @xcite or abstractive @xcite . although the tasks of slide and poster generation have generally been considered separate from scientific summarization, they are related in that both tasks require taking an input article, then organizing and abstracting the information to generate a new document. past work has developed methods for slide generation from papers @xcite @xcite , from code @xcite , or based on a query @xcite . poster generation has been explored in the form of content extraction for posters (xu and wan, 2021), interactive generation @xcite , or full content generation using graphical models @xcite . to the best of our knowledge, our work is the first to create a unified method capable of generating a diverse range of templatic views of a source document. large language models (llms), which are central to our approach, have shown impressive capabilities in a variety of tasks @xcite . based on the transformer architecture @xcite , llms have shown emergent abilities in tasks such as arithmetic and question answering @xcite . similar to chain of thought prompting @xcite and content planning prompting @xcite , we show that by generating an intermediate representation of an input document can improve performance over simply prompting the model to generate the final document from the original input. as past work has tackled generation of templatic views as separate tasks, methods for automatic evaluation of different document types is fragmented. longsumm, the shared task introduced by @xcite , uses rouge to evaluate model performance @xcite . @xcite introduced slide level rouge to evaluate slide generation, a variant that contains a penalty for the number of slides. @xcite used a trained regressor. for summarization, many automatic evaluation metrics have been introduced such as bertscore @xcite , unieval @xcite , bartscore @xcite , blanc @xcite , and moverscore @xcite . however, these metrics are intended for a simple input documentsummary setup, and do not take into account factors that affect the quality of other types of documents (e.g. structure). our work is the first to introduce template adaptable evaluation, allowing uniform comparison of performance across template types. ## data we begin by describing the data used in this paper. there is no existing dataset that includes multiple views of a single document. instead, we evaluate our unified method, described in §4, on 3 existing datasets: doc2ppt, longsumm, and paper-poster @xcite @xcite . these datasets are chosen because they each involve generating a different view of a document. although our method is not specific to the scientific domain, it is one of the few domains with abundantly available public data of multiple templatic views foot_1 . the three datasets and their associated generation tasks are described below. slide generation. we use use the doc2ppt dataset @xcite , which contains 5.8k scientific papers in computer science and their respective slide decks. as @xcite do not release data splits or code, we randomly sample 1k examples from this dataset for evaluation. the slides are provided as an image for each slide. we use the azure ocr tool to extract the text from each slide foot_2 . ## unified llm-powered generation of templatic views the most straightforward way to transform documents between templatic views using llms, is to simply prompt the system to generate the target view given the input. however, similar to chain of thought prompting @xcite , we hypothesize that first generating a structured, intermediate representation of an input document and then reasoning over that representation will result in better generations than directly prompting the model. our goal is to evaluate the capabilites of llms to generate long, structured documents, and experiment with how structured prompting can improve performance. we experiment with a simple general two-step process: first generate an intermediate representation, then generate the templatic view. these steps are described in greater detail below, and the process is visualized in figure 1 . ## template adaptable evaluation prior work on document generation has treated the evaluation of different templatic views as separate tasks. thus, our goal is to develop a framework of automatic evaluation that is template adaptable. this not only allows us to compare performance across diverse datasets, it also removes the requirement of designing and maintaining individual metrics for each template. in order to generalize to multiple templates, we introduce the concept of panels. a panel is a unit of organization within a document type, for which the placement and ordering of the panel is important to the overall flow of information in the document. for example, we consider panels to be each slide in a slide deck and each section on a poster. we consider the entirety of a blog post to be a single panel. although we test our method on the tasks of slide, blog, and poster generation, the concept of panels is not limited to these document types. for example, each post on a social media thread could be considered a panel, or each page on a website. we aim to unify the evaluation of templatic views by integrating prior metrics into a template adaptable precision-recall framework, which we refer to as template-adaptable evaluation (tae). tae is not a new individual metric, but rather an evaluation framework that allows generalization to new templates. for example, tae can even be used with rouge to evaluate poster generation. the general tae formulation is as follows: in which q p is the precision measure of quality ( §5.1), o p is the precision penalty for order ( §5.2), and l is the non-reflexive penalty for length ( §5.3). similarly, q r is the recall quality measure and o r is the recall penalty for order. the precision-recall formulation allows evaluators to decide which mea-sure is most important to them, or calculate an overall f-measure score. ## results as mentioned in §3, past work on doc2ppt and paper-posters do not release code, making it difficult to do a direct comparison. they also do not report any baselines to compare against. meanwhile, longsumm's blind test does not allow us to compute our custom metric, although we do report the leaderboard results in appendix c. notably, with almost no prompt engineering our llm-based system places second on this leaderboard. we argue that for the investigation in this paper, direct comparison to prior non-llm baselines is not only unfair to those approaches, but not particularly insightful. therefore, similar to wei et al. (2022b), we focus on variants of our llm-based method and treat them as baselines. example outputs of each template type can be found in appendix d. we conduct experiments with the following settings: (1) no representation -this is the default setting of going directly from the source document to the target document. we skip the intermediate generation step, passing the full paper as input. we experiment both with and without the style parameters. (2) own representation -we do not pass a json structure to the intermediate generation similarity measure rep. style r-l m b berts slides none × 5.0 6.4 0.3 31.6 none ✓ 5.1 6.0 0.4 31.7 own ✓ 6.5 7.1 1.2 36.1 text ✓ 7.3 8.0 1.4 36.4 json × 4.2 6.0 0.3 31.4 json ✓ 7.4 8.4 1.5 36.9 blogs none × 26.6 19.6 3.0 82.5 none ✓ 25.1 17.7 2.3 82.8 own ✓ 23.9 19.2 2.3 82.2 text ✓ 25.4 19.3 2.5 82.5 json × 28.3 25.3 5.0 82.3 json ✓ 25.4 19.6 2.8 82.4 posters none × 8.1 10.3 1.0 35.6 none ✓ 10.1 11.6 1.9 39.5 own ✓ 12.8 12.6 2.9 52.8 text ✓ 11.3 11.7 2.1 45.9 json × 14.2 16.8 4.0 52.8 json ✓ 15.5 14.5 15.3 53.3 for each template, we experiment with different representations (rep) and whether or not we include the style parameters (style). we report the tae f1 scores as calculated in §5, using rouge-l (r-l), meteor (m), bleu (b), and bertscore (berts) as the similarity metrics. we use gpt35-16k-0613 in our main set of experiments. we truncate text that is too long for the input window and use a temperature of 0.0 as standard. 5 ## conclusion in many domains, people choose to disseminate information across different modalities and formats for better communication to broader audiences. we proposed a unified view of document transformation and evaluation. we showed that llms are capable of templatic document generation with minimal supervision, and that a structured, intermediate representation can improve performance, particularly for smaller models. we also introduced a flexible precision-recall framework for automatic evaluation that easily integrates existing evaluation metrics into a unified system and allows for comparison across diverse datasets without additional task specific metric design. finally, we conducted a human evaluation and showed that annotators prefer the documents generated using the intermediate representation 82% of the time and that our evaluation framework correlates better with human preference than standard evaluation metrics.