STRUCTSUM Generation for Faster Text Comprehension (2024)

Abstract: We consider the task of generating structured representations of text using large language models (LLMs). We focus on tables and mind maps as representative modalities. Tables are more organized way of representing data, while mind maps provide a visually dynamic and flexible approach, particularly suitable for sparse content. Despite the effectiveness of LLMs on different tasks, we show that current models struggle with generating structured outputs. In response, we present effective prompting strategies for both of these tasks. We introduce a taxonomy of problems around factuality, global and local structure, common to both modalities and propose a set of critiques to tackle these issues resulting in an absolute improvement in accuracy of +37 pp (79%) for mind maps and +15 pp (78%) for tables. To evaluate semantic coverage of generated structured representations we propose Auto-QA, and we verify the adequacy of Auto-QA using SQuAD dataset. We further evaluate the usefulness of structured representations via a text comprehension user study. The results show a significant reduction in comprehension time compared to text when using table (42.9%) and mind map (31.9%), without loss in accuracy.

## introduction the overwhelming amount of information available online poses a significant challenge for users seeking to quickly grasp and process relevant information. current large language models (llms), such as palm-2 (palm2, 2023), gemini @xcite and @xcite , while capable of providing text-based responses to user queries, often fail to adequately structure and organize this information in a way that facilitates comprehension @xcite . this can lead to information processing bottlenecks that hinder * work done while interning at google deepmind. input text text 1 divide prompt text 2 text n text to table prompt input text root generation expansion the input passage is initially segmented into subpassages, followed by the generation of multiple tables. @xcite demonstrates the generation process for mind maps. after the main concept has been generated, an iterative expansion phase ensues, during which the mind map is expanded until termination. to address this issue, we introduce the notion of structured summaries, or structsum in short. structsums are derived by hierarchically organizing information and inducing semantic connections from an input text passage. without loss of generality, we focus on tables @xcite and mind maps @xcite as possible structsum instantiations: through extensive experimentation with palm-2 (palm2, 2023), we show that llms are not always effective at generating structsums that are factual and structurally correct. to overcome these issues we propose a pipeline for structured data generation. our pipeline consists of structure-specific prompts followed by critics to assess output quality along three different dimensions, that are common both to tables and mind maps: (i) factuality, (ii) local structure and (iii) global structure. we found that our proposed critics improved the overall quality of the generated output by +37pp for mind maps and +15pp for tables. to ensure the usefulness of structsum for text-comprehension tasks, we propose auto-qa as a measure of output coverage. we automatically generate qa pairs from input text and use structured outputs to answer these questions. furthermore, we verify the appropriateness of using auto-qa by comparing auto-qa with human generated qa pairs on squad @xcite development set. finally, starting from the initial hypothesis that structsums can enhance the effectiveness of information-seeking scenarios, we conducted a user study to evaluate their impact on users' ability to process information, using a text comprehension user study. results demonstrate how struct-sums improve information seeking, specifically on timed text comprehension metrics. we found that by using the structured representation, users can answer questions 42.9% faster for tables and 31.9% for mind maps. displacement length beam draught 4,050 long tons 300 feet 46 feet 20 feet 2 inches speed range complement -18 knots 8,750 nautical miles 300 to 350 officers and ratings armament ---two breech-loading (bl) 8-inch (203 mm) guns, one each… armour --- a lower armoured deck that was 2 inches (51 mm) on the flat and 3 inches... ## generating structsums we focus on tables and mind maps as a possible structsum instantiations. ## data generation pipeline we now present our structsum data generation pipeline. although each structsum is seemingly different, we identify three dimensions that are common to both table and mind map modalities: (i) factuality, (ii) local structure, and (iii) global structure. we use a set of critics, implemented via prompts, to ensure sufficient quality across each dimension. through our initial experiments we find that tweaking each critic according to the structure is more helpful. ## model for all the experiments, we use the unicorn (palm-2-unicorn, 2023) variant of palm-2, a fine-tuned transformer-based model with ul2 @xcite like objectives. palm-2 improves on palm @xcite through optimized scaling, richer training data and instructing tuning @xcite . ## dataset to test our pipeline on a diverse set of input passages, we selected wikipedia text as the source. specifically, we started with the english split of the wiki40b @xcite ) dataset foot_2 . the dataset is cleaned up by page filtering to remove disambiguation pages, redirect pages, deleted pages, and non-entity pages. input to our prompts are passages that are obtained by splitting the wikipedia text using the _start_paragraph_ symbol that is already provided as part of the dataset. ## results in this section, we present the results of our experiments using palm-2. ## data generation statistics ## limitations of auto-qa we propose auto-qa as a coverage metric, in order to measure how much information from the original passage is retained in the structsum. note that the metric is not intended as a substitute for evaluations of overall quality. the primary benefit of auto-qa is the synthetic question generation component, which can be run at scale without costly human annotations. in ## conclusion in this work we study the potential of structured representations like tables and mind maps to enhance information comprehension. utilizing our divideand-generate prompting and iterative expansion, we achieved significant improvements in output quality (+37pp for mind maps, +15pp for tables) using structure-specific prompts and critics. we proposed auto-qa based coverage metric that automatically generates qa pairs from the input text and uses structsum outputs to answer them.