Zooming in on Zero-Shot Intent-Guided and Grounded Document Generation using LLM s (2024)

Abstract: Repurposing existing content on-the-fly to suit author’s goals for creating initial drafts is crucial for document creation. We introduce the task of intent-guided and grounded document generation: given a user-specified intent (e.g., section title) and a few reference documents, the goal is to generate section-level multimodal documents spanning text and images, grounded on the given references, in a zero-shot setting. We present a data curation strategy to obtain general-domain samples from Wikipedia, and collect 1,000 Wikipedia sections consisting of textual and image content along with appropriate intent specifications and references. We propose a simple yet effective planning-based prompting strategy, Multimodal Plan-And-Write (MM-PAW), to prompt LLMs to generate an intermediate plan with text and image descriptions, to guide the subsequent generation. We compare the performances of MM-PAW and a text-only variant of it with those of zero-shot Chain-of-Thought (CoT) using recent close and open-domain LLMs. Both of them lead to significantly better performances in terms of content relevance, structure, and groundedness to the references, more so in the smaller models (upto 12.5 points increase in Rouge 1-F1) than in the larger ones (upto 4 points increase in R1-F1). They are particularly effective in improving relatively smaller models’ performances, to be on par or higher than those of their larger counterparts for this task.

## introduction recent advances in generative models @xcite @xcite have enabled the creation of high-quality textual and visual content through natural language prompts. techniques like chainof-thought (cot) @xcite ., † work done while interning at adobe research ‡ work done while working at adobe research 2023) have improved llms' performance across nlp tasks, including question answering @xcite , reasoning @xcite , summarization @xcite , and conversation generation @xcite . document creation can be a creative process; while the content itself may or may not always be unique, the goal or intent of each document can be very specific to the user's needs. it typically involves reusing and piecing together portions of content from multiple sources to create a rich first draft based on the intent, and then iteratively edit it until it reaches a suitable final stage. figure 1 illustrates this scenario of creating a wikipedia section; the author aims to create a first draft for a specific section using a few reference articles. in such scenarios, zero-shot generation of first draft can provide a strong starting point, and save the time and effort of content creators creating general-domain documents such as marketing blogs, reports, etc. in this paper, we introduce intent-guided and grounded long document generation in zero-shot setting, with three constraints: (i) documents are to be generated from the given reference documents and an intent specified by the user; (ii) documents can be multimodal in nature with text and image content; and (iii) the generation is to be on-thefly for any given intent with a few source documents and no additional training data. we present a data curation strategy to obtain general-domain wikipedia samples, and curate an evaluation set comprising of 1,000 sections along with the corresponding intents and external references using xml parsing and bing search. grounding has been a well-known paradigm in natural language generation wherein some source content is used to condition the generation @xcite . however, most grounded generation works focused on short texts @xcite , whereas our focus will be on long documents ranging over several sentences. further, most document generation works are limited to text-only generation; while text-to-image models @xcite like dall-e generate highquality images from textual prompts, automatically determining the appropriate textual and visual composition of a document based on an intent and references remains underexplored. inspired by the superior performances of llms in zero-shot settings @xcite , we propose a zero-shot prompting strategy that infuses content planning as an intermediate step in the generation task. our pipeline comprises of a retriever module to retrieve the relevant content from the given references based on the intent, followed by an llm prompting module to plan and synthesize the output. specifically, we propose multimodal plan-and-write (mm-paw) prompting, to generate multimodal plans comprising of text topics and image-specific descriptions, based on intent and retrieved content, and condition the text generation on the generated plan. appropriate images are generated using image descriptions using text-to-image models. we compare mm-paw and a text-only variant of it (paw) (for multimodal and text-only section generation respectively) with zero-shot cot using 8 close and open-source llms. we note improvements using our prompting variants in terms of content relevance and coverage while maintaining groundedness. specifically, they improve the smaller models' performances to be on par with or higher than those of their larger counterparts, indicating the effectiveness of our approach in utilizing smaller models to perform the task comparable to the larger ones. to our knowledge, this is the first study on grounded multimodal document generation using llms. plan-based generation. content planning has been a widely studied topic in natural language generation tasks @xcite @xcite , as they assist in enforcing coherence, structure, and logical consistency for longer text generation. @xcite addressed paragraph completion by first predicting key words for the missing content, and using them to generate the sentences. @xcite focussed on planning a sequence of events using event graphs to guide a story generator. @xcite use ordered sequences of entities to ground the summary generation. more recently, planning-based approaches to better prompt large language models have been gaining attention @xcite @xcite . wang et al. (2023a) proposed zero-shot plan-and-solve prompting for multi-step reasoning tasks. wang et al. (2023b) used planning in summarization using llms, by first prompting them to answer a few elemental questions and using them to generate the summaries step by step. we extend the concept of planning to prompt llms in a zero-shot manner to generate multimodal plans providing cues on the preferred textual and visual composition of output, and ground the subsequent generation on them. ## task setup & dataset writer's block is a major challenge for content creators, which can affect their productivity and creativity while creating new content. however, document creation seldom starts from scratch, and obtain rough first drafts and revising them can enhance the writing abilities of creators @xcite . we study the task of automatically providing a rich multimodal first draft that aligns to author's goals, while reusing relevant information from across different related sources, which they can further iterate upon to create their final version. we study this task in a zero-shot setup without any fine-tuning, and investigate the capabilities of llms to generate content on any given topic provided a few references to it. there do not exist datasets tailored for our task. we find wikipedia as the most suited source due to the following reasons: (i) we can view the various section titles as intents, and the citations can act as the external references to create a given section; and (ii) wikipedia articles have text and image content, where the images contain content related to specific concepts in the text. wikipedia is increasing being used as a source for various tasks @xcite ; however, they do not provide multimodal articles with images along with text. ## method our pipeline follows the retrieve-and-generate paradigm @xcite and consists of two stages, namely intent-guided content extraction and document generation (figure 2 ). in the first stage, we perform query-based sentence retrieval to extract relevant sentences from the reference articles, using the given intent (section title) as the query. we use sbert embeddings @xcite to encode reference sentences and employ faiss @xcite to perform fast semantic search by indexing these embeddings. we compute the similarity of the intent with the indexed sentences, and the top-k sentences are selected. in the second stage, we incorporate the intent and the retrieved sentences in our zero-shot prompt template namely multimodal plan-and-write (mm-paw) to prompt an llm. the order of retrieved sentences in the prompt is in order of semantic similarity (cosine similarity) with the given intent. ## experiments we conduct our experiments using two closesource and two open-source family of llms, namely claude (claude-3-haiku) (anthropic, 2024), gpt (gpt-4, gpt-35-turbo) @xcite , llama (fine-tuned chat 70b, 13b, 7b models) @xcite , and mistral (7b, 8x7b) @xcite . we use nvidia a100 gpus to perform inference with the llama and mistral variants. for intent-based sentence retrieval, we set @xmath0, so as to accommodate for the context length limits of llama and mistral models. 4 we use the stable-diffusion-v1-5 checkpoint @xcite to generate images. in order to have a fair comparison, a length constraint is enforced in the prompt template so as to ensure that the generation and the ground truth are of similar lengths. the expected range of words to be produced is defined as [0.8, 1.2] times the number of words in the ground truth. results are averaged across 5 runs with different seeds. standard deviation of the runs are provided in appendix d. baselines. the instructions to the llms are minimal in the baseline setup. the llms are prompted to generate coherent section text using the intent and retrieved sentences along with the length specification. the intent itself used as the text prompt to generate images using the text-to-image model. the baseline prompt is provided in appendix b. ## results & discussion on an average, the improvements of our variants over the baselines in terms of text coverage (recall) are higher than those for relevance (precision). given the retrieved sentences as input, we believe the baseline models' selection of relevant details may not result in a good coverage of relevant topics. this challenge arises from the complex and underspecified dependency between a short intent (the section heading) and retrieved reference sentences, making it more challenging for language models to accurately capture, as highlighted in @xcite . our proposed approach formulates a high-level topic-based plan first, providing the model with an intermediate overview of the references' diverse aspects, thereby increasing coverage. interestingly, between paw and mm-paw, we note that the former has higher coverage and overall scores for text generation, while mm-paw has slightly higher relevance values. we speculate that including image-specific details in the multimodal plan may have made the topics more "accurate" and aligned with the themes in the image descriptions, thereby increasing relevance (and reducing coverage). we believe further investigation would help in understanding the interplay between the text and image quality while generating multimodal content and text-only subset of it. in terms of groundedness and structure, our proposed variants result in improved scores for both nli and g-eval in most cases, indicating our outputs are more grounded to the references compared to baselines. finally, the images generated using the multimodal plans result in more relevant outputs as seen in the clipscore compared to using only the intent or high-level section details with the baseline approaches. we note that the improvements over the baselines are higher in larger model variants (such as claude, gpt-4, and gpt-3.5) compared to the smaller ones, indicating their su-gt vertebrates originated during the cambrian explosion, which saw a rise in organism diversity. the earliest known vertebrates belongs to the chengjiang biota and lived about 518 million years ago. . . these groups had the basic vertebrate body plan: a notochord, rudimentary vertebrae, and a well-defined head and tail. all of these early vertebrates lacked jaws in the common sense and relied on filter feeding close to the seabed. . ., small eel-like conodonts, are known from microfossils of their paired tooth segments from the late cambrian to the end of the triassic. baseline vertebrates emerged during the cambrian explosion. these primitive vertebrates likely possessed simple skeletal structures . . . the feeding habits of early vertebrates were . . . the earliest known vertebrates lacked conventional jaws, relying on filter feeding near the seabed. although limited fossil evidence makes it difficult to precisely determine their physical characteristics,. . . ## conclusions we introduced the task of intent-guided and grounded long document generation, and planningbased prompting strategies to come up with multimodal plans while generating documents. we curate a dataset for evaluating our approaches with respect to baseline llms by scraping multimodal