S ci D oc2 D iagrammer- MAF : Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement (2024)

Abstract: Automating the creation of scientific diagrams from academic papers can significantly streamline the development of tutorials, presentations, and posters, thereby saving time and accelerating the process. Current text-to-image models (Rombach et al., 2022a; Belouadi et al., 2023) struggle with generating accurate and visually appealing diagrams from long-context inputs. We propose SciDoc2Diagram, a task that extracts relevant information from scientific papers and generates diagrams, along with a benchmarking dataset, SciDoc2DiagramBench. We develop a multi-step pipeline SciDoc2Diagrammer that generates diagrams based on user intentions using intermediate code generation. We observed that initial diagram drafts were often incomplete or unfaithful to the source, leading us to develop SciDoc2Diagrammer-Multi-Aspect-Feedback (MAF), a refinement strategy that significantly enhances factual correctness and visual appeal and outperforms existing models on both automatic and human judgement.

## introduction researchers often need to transform detailed studies into compelling diagrams for presentations, going beyond the usual figures in their papers to include summaries, methodologies, and dynamic data visualizations @xcite . 1 automation of diagram generation from the multimodal components of academic papers-such as text, figures, tables, and plots-could enable the creation of accurate and appealing visual aids for scientific tutorials and presentations @xcite @xcite , which can help researchers and educators to improve productivity and enhancing the overall quality of their presentations. we introduce a new task sci-doc2diagram where the input comprises of the content of academic papers and user's intent and 1 according to @xcite , approximately 39.2% of figures in acl slides are creatively derived from the original papers, taking forms like flowcharts, summary tables, and plots/charts that visually interpret the paper's content. the output is a scientific diagram crafted from the content of papers and aligned with user's intent, which we call "extrapolated-diagram". consider a scenario where a busy scientist wants to create figures from papers within a tight deadline, with the specific goal of generating a bar chart as shown in the figure 1 . to achieve this quickly, recent advancements in text-to-image (t2i) generation models like stable diffusion @xcite and dall-e @xcite offer promising solutions. however, these models face limitations in generating factually correct and visually appealing scientific figures. they primarily produce raster graphics at low resolutions, which is not suitable for scientific figures requiring precision and valid text @xcite . @xcite introduced a scientific diagram generation method based on only user's intent. however, creation of informative diagrams using long-content papers as an additional input remains unexplored. how effective are the current state-of-the-art large language models (llms) or visual language models (vlms) at generating scientific diagrams? despite their potential, a significant knowledge gap exists: there is currently no standard benchmark to assess their performance in this area. to address this gap, we introduce the sci-doc2diagrambench dataset that features 1080 diagrams from 89 *acl papers, sourced from existing slides in acl anthology (section 3). this dataset supports researchers in automating scientific diagram creation for presentations. we evaluate state-of-the-art llms @xcite @xcite to extract relevant information from documents pertinent to the user-specified intent. specifically, we use a multi-step pipelined method, scidoc2diagrammer that first extracts relevant data from academic papers corresponding to intent of the user in creating the diagram, generates an intermediate code that renders the final diagram (section 4.1) as shown in figure 1 . however, the first-draft images generated by these models suffer from a range of errors: hallucination, incompleteness and unreadability due to their incapabilities of parsing long-prompts @xcite . therefore, we propose a sequential refinement approach scidoc2diagrammer-maf based on the intial draft generated by scidoc2diagrammer. the new approach is guided by several critic models to provide targeted feedback (multiple aspects such as completeness, faithfulness, and look and feel) in a step-by-step hierarchical manner on the output to further enhance the quality of output (section 4.2). our multi-aspect feedback refinement approach significantly improves diagram quality, as confirmed by both automatic and human evaluations, demonstrating notable enhancements over existing methods (section 6). complex flowcharts and tables derived from single or multiple documents show notable enhancements in faithfulness and completeness when refined sequentially. it is particularly advantageous when iteratively synthesizing information from multiple sources to enhance clarity and precision. 2 2 task formulation of scidoc2diagram our goal is to automatically generate scientific diagrams from academic papers tailored to user specifications. we process the input document @xmath0, f, τ, π}, which includes body text t , figures f , tables τ , and plots π, guided by a user intent i (e.g., "create a bar chart showing the comparison..." in figure 18 ). the output, an extrapolated-diagram δ, visually represents content not explicitly illustrated in the paper. ## scidoc2diagrambench: a new dataset creating extrapolated-diagram from source papers is an important yet challenging task. however, no current dataset evaluates state-of-the-art models on this task. to advance research, we have created scidoc2diagrambench dataset, repurposing doc2ppt @xcite (scientific papers and corresponding slides from acl anthology) and tutorialbank @xcite (tutorials from *acl conferences). 3 our dataset includes the following three parts and the overall statistics of different diagram types are in scidoc2diagrambench-gold. our initial goal was to explore what are the types of extrapolated-diagram usually created by humans while making presentations. we hired an expert annotator from upwork with more than five years of authoring nlp-related papers and creating presentations. the annotator had two tasks, first to identify slides containing extrapolated-diagram (excludes slides that used external information not found in the paper), second to manually write an intent for the corresponding diagram creation (13, 12). through this annotation process, we collect 265 gold-standard extrapolated-diagram from 34 papers where each instance comprises of a tuple @xmath1, intent of diagram, gold dia-gram>. the first three authors manually checked the intents, and only instances that were unanimously deemed reasonable were retained. after analyzing these diagrams, 32% were flowcharts illustrating methodology (extrapolated-flowchart), 33% were results/plots (extrapolated-results), 25% were summary of related works/contributions (extrapolated-summary) and 10% were related to clarifying existing architecture (extrapolated-architecture). table create a bar chart showing the comparison of the precision of nash on reuter dataset with that of the three best performing baselines on 128 bits along with providing short description of each of the models question generator data store qa pairs obtained (diagram plan) source pdf \cite{shen-etal-2018-nash} completeness evaluator faithfulness evaluator (compares answers of questions from both pdf and image) cq1: is it a bar chart? cq2: is the comparison between 4 models shown? cq3: is there a short description of each of the models? …… fq1: what are the three best performing baselines? fq2: what are the performances corresponding to the three best performing baselines? …… fq1: what are the three best performing baselines? fq2: what are the performances corresponding to the three best performing baselines? …… -show only the comparison among 4 best performing models -include short description in the axes of each model refiner after comparison layout evaluator + refiner scidoc2diagrammer scidoc2diagrammer-maf stage 1: diagram planning stage 2: diagram rendering scidoc2diagrambench-extended. using insights from scidoc2diagrambench-gold on how presenters transform paper content into diagrams, we used gpt-4 to generate new diagram intents for this subset of data ( ## methodology the scidoc2diagram task comprises of generating initial first-draft generation of diagrams from documents using base models (section 4.1), followed by refinement of diagrams using critic modules (section 4.2). ## results and findings our main objective is to explore the creation of 'extrapolated' scientific visuals from documents using zero-shot/few-shot settings of advanced vlms, and also analyze whether the quality of these visuals without dataset-specific fine-tuning, can be improved using our proposed scidoc2diagrammer-maf. to answer this broad question, we focus our experiments on answering the following research questions as follows: rq2: how effective is our scidoc2diagrammer-maf over scidoc2diagrammer? based on drawing inspiration from ribeiro et al. (2020), we acknowledge that automatic metrics alone are insufficient to fully evaluate model performance, so we also conduct human evaluation. for scidoc2diagrambench gold and scimultidoc2diagrambench-gold, we had human raters evaluate the quality of diagrams produced by different models and compare these to actual human-created slides, considering the source paper and the diagram's intended purpose for 30 randomly sampled diagrams. in the scimultidoc2diagrambench-extended, which lacks a gold standard image, only the paper, the intent of the diagram, and the model-generated images were presented for evaluation on 50 randomly sampled images. two authors and two professional crowdworkers from upwork assessed the quality of all images, using criteria such as completeness, faithfulness, and layout from 1 to 5. figure 4 illustrates that diagrams generated using our refinement strategies compared to few-shot and self-refine strategies consistently receive the highest scores from humans on all dataset subsets. rq3: what type of refinement works the best across each diagram type based on automatic evaluation? from rq4: is refinement more effective on simple images or complex images? across three datasets, the multi-aspect refinement strategy consistently outperforms the few-shot approach in diagram production (figure 4 ). on the simplest dataset, scidoc2diagrambench-gold, maf-refine performs significantly better than few-shot approach, with improvements of 0.9 in completeness, 1.1 in faithfulness, and 0.8 in layout. as complexity increases in scidoc2diagrambench-extended, maf-refine still performs strongly, showing improvements of 1.4 in completeness, 1.0 in faithfulness, and 0.5 in layout. even in the most complex dataset, scimultidoc2diagrambench-extended, maf-refine leads with significant improvements, showing that maf-refine significantly boosts diagram quality across datasets and maintains its benefits even as complexity rises. ## discussion and future work we introduce an innovative task for constructing scientific diagrams from scientific document discourse, commonly used in presentations/posters. we establish a new benchmark using document- ## effectiveness of our diagramgeneration pipeline based on the survey responses in 9, we selected the paper requiring the most number of "extrapolated-diagrams" and asked two participants from the set of 11 people (control group) to create a flowchart and a bar chart from scratch after reading the paper content. on average, it took them 30-35 minutes per diagram. we then provided two other participants (experimental group) with access to our interactive scidoc2diagrammer-maf system to create the same set of diagrams (a flowchart and a bar chart). with our tool, they completed each diagram in 8-10 minutes. we also provided intermediate questions and answers to extract data from the paper (figure 8 ), code (figure 9 ) which helped reduce the cognitive load by simplifying the process of diagram creation. the participants found it easier to tweak a template diagram generated through code, instead of searching for the relevant data from paper, and writing a code from scratch to generate figures (sample shown in (figure 10 )). ## instruction guidelines on annotating and identifying extrapolated diagrams objective: you are provided with a dataset containing scientific papers and their corresponding slide decks. your task is to annotate each slide in the paper according to predefined categories based on the content origin and presentation style. ## instructions on annotating intent of the diagram creation objective: your role is to provide clear, comprehensive, and complete descriptions for diagrams in scientific slide decks, focusing on their intent and content. this will enable any individual to understand or recreate the diagram based purely on your description. ## addressing data leakage concerns we address the potential data leakage due to the use of existing datasets such as doc2ppt @xcite and tutorialbank @xcite in constructing the scidoc2diagrambench by clarifying that our task is fundamentally distinct from these datasets' original objectives, such as slide creation. specifically, scidoc2diagram aims to evaluate large language models' (llms) ability to generate scientific diagrams conditioned on user intent and long-document context, which presents a more nuanced and challenging task than the straightforward generation of slides from text. our task focuses on creating scientific diagrams based on detailed user instructions and content from the academic paper itself. this involves synthesizing complex ideas into visual form, which is substantially different from simply converting scientific text into slides. as such, the focus shifts from summarization or slide creation to a more advanced task of diagrammatic representation, ensuring that any overlap with doc2ppt @xcite or tutorialbank @xcite is minimized. additionally, no direct train-test splits are used, and our dataset is crafted to assess the ability to generalize across varied diagrammatic intents. besides, it is worth noting that many early benchmarks in the field have been developed by reusing existing datasets or open-source repositories to create new tasks. examples include works like automatikz @xcite , personad2s @xcite , and e2ekg @xcite which repurpose existing data to establish more specific, challenging benchmarks. our work follows a similar approach, using existing resources to create a more complex benchmark aimed at evaluating how diagrams can be generated not only based on the academic paper but also in line with the specific user intent, a dimension absent in doc2ppt and tutorialbank. to further mitigate concerns about data leakage and to enhance the diversity and representativeness of the scidoc2diagrambench, we have incorporated expert human annotations. these annotations include additional metadata such as the intent behind the created diagram. the inclusion of this metadata makes each instance more unique and specifically tailored to the task of diagram creation, thereby distinguishing it from simple text-to-slide generation datasets. this human-curated layer, not present in doc2ppt or tutorialbank, ensures that our dataset adds substantial value and differentiation from prior resources. in conclusion, while we leverage existing datasets for constructing scidoc2diagrambench, the way we have formulated the task introduces a significant layer of complexity and novelty. by focusing on user intent, incorporating human annotations, and avoiding standard train-test splits, we ensure that scidoc2diagrambench remains a challenging and representative benchmark that builds on existing resources while addressing concerns about potential data leakage. besides, a significant number of early researchers have created benchmarks by reusing existing datasets or open-source repositories such as automatikz @xcite figure 10 : displays a feedback form designed to allow users to evaluate and improve the generated diagram and its underlying code. users can provide ratings for the completeness and correctness of the diagram on a scale from 1 to 5. additionally, users can ask clarification questions to verify the completeness of the generated content by submitting their queries in a comma-separated format. they can also input correct data or corrections in the text box provided to regenerate the code. the interface includes buttons to either "regenerate" or "save" the revised code based on user input. this tool enables interactive refinement of generated diagrams based on user feedback. 2023), personad2s @xcite , endto-end scientific knowledge graph construction @xcite , vqa @xcite . 15.2 examples from scidoc2diagrambench-gold figure 15 (extrapolated-flowchart) corresponds to intent "construct a flowchart or diagram showing figure 16 (extrapolated-results) corresponds to intent "create a bar chart showing the comparison of f1 score of morse, morfessor s+w, morfessor s+w+l and morphochain against published stateof-the-art results" and the source paper is @xcite . types intent source paper extrapolated-flowchart create a flowchart to explain the process of how the proposed model understands the semantics of problems and decides which symbol to generate next." (chiang and chen, 2019) extrapolated-summary create a table to summarize related works in sarcasm detection, highlighting the novelty of the proposed model in terms of its ability to model contrast and incongruity (tay et al., 2018) extrapolated-results convert the table describing the different translations of titles into a bar chart, grouping each translation by its source (headline, lead, editor, opennmt, hybridfusion), and using color coding to highlight differences in the length and tone of each translation (murao et al., 2019) ## additional experiments individual components are necessary in sci-doc2diagrammer! ## comparison with contemporary research on code generation unlike chartmimic @xcite and mm-code @xcite , which primarily focus on either visual charts or programming challenges as input, our approach uses the high-level intent provided by the user to generate complex scientific diagrams. this makes our work distinct as it emphasizes understanding longer document contexts and user-specific requests rather than predefined visual or mathematical data. existing works such as mmcode @xcite and design2code @xcite operate on well-defined inputs like programming challenges or website designs. in contrast, our approach supports the processing of long scientific figure 16: an example from scidoc2diagrambench-gold where the intent is "create a bar chart showing the comparison of f1 score of morse, morfessor s+w, morfessor s+w+l and morphochain against published state-of-the-art results" and the source paper is @xcite documents, capturing intricate diagram intent from user-specified slides or text sources, making our method applicable for use in research papers and presentations. while chartmimic and mmcode focus on gen-erating code that addresses visual understanding or mathematical problems, our method generates scientific diagrams for seamless integration into academic presentations. this involves translating user intent into structured diagrams that comply with okay, let me "create a flowchart showing the methodology of how similar context is used for human and model response generation and the way it has to be evaluated" (intent) oh no! i only have an hour before my presentation. how can i quickly create factually correct and informative visuals from the papers i need to present? you should first try out the existing text-to-image models with a textual description as your prompt source pdf stable diffusion automat ikz scidoc2diagr ammer w/ gpt4 scidoc2diagrammer-maf include equations that need to be present for showing evaluation -show clear input and outputs from humans and models -precision and recall equations have to be included -arrows should be directed in nature -there should not be overlapping nodes -there should not be arrow markers inside the node c f l figure 18: a busy scientist is inclined to automate the creation of diagrams, where our refinement-based approach in scidocdiagrammarmaf yields a complete and faithful diagram. in addition, human-computer interaction is an emerging topic as artificial intelligence (ai) advances in diverse tasks. however, ai still faces challenges such as gender and race discrimination @xcite , producing unfaithful information through hallucinations @xcite , privacy breaches @xcite , etc. fully relyalgorithm 1 completeness assessment (c critic ) require: o: diagram to be evaluated, code: associated code to be checked require: p df : document source require: i: defined intent for diagram creation ensure: c: completeness score of o and c f eedback to improve the score 1: @xmath4: decompose i and generate a set of questions q based on the decomposed intent 3: for each question q i in q do 4: a i ← extract(q i , o) ▷ get answer determining completeness of o 5: s i ← [a i is adequately represented in o] ▷ assign score from 1-5 6: append s i to score list s 7: c score ← 1 |q| ∑︁ |q| @xmath5: c f eedback ←textual feedback to improve c 9: return c score and c f eedback require: d: diagram to be evaluated require: r: design rules for layout and aesthetics ensure: s: aesthetics score of diagram d ensure: f : feedback for improving d 1: function evaluatelayout(d, r) 2: s ← 0 3: f eedback ← [] 4: for each rule r in r do 5: result, comment ← applyrule(r, d) 6: s ← s + result 7: f eedback.append(comment) 8: return s, f eedback algorithm 2 faithfulness assessment (f critic ) require: o: diagram to be evaluated and corresponding code code require: p df : document source require: i: defined intent for diagram creation ensure: f : faithfulness score of o and f f eedback to improve the score 1: using o and code, generate questions that would validate the correctness of o 2: @xmath6: for each question q i in q do 4: a pdf i ← retrieve(q i , p df ) ▷ retrieve answer from pdf 5: a image i ← extract(q i , o) ▷ retrieve answer from diagram o 6: s i ← [a image @xmath7: append s i to score list s 8: f score ← 1 |q| ∑︁ |q| @xmath8: f f eedback ←textual feedback to improve c 10: return f score and f f eedback algorithm 3 sequential refinement of diagrams require: diagram d, source pdf p ensure: refined diagram d ref 1: for type in {'completeness', 'faithfulness', 'layout'} do 2: maxiterations ← 3 3: threshold ← 4.5 4: iteration ← 0 5: score ← 0 6: while @xmath9: feedback ← evaluate(d, p , type) ▷ evaluation based on type 8: if score(feedback) > threshold then 9: break ▷ exit loop if score exceeds the threshold 10: d ← refinediagram(d, feedback) ▷ refine diagram based on feedback 11: score ← score(feedback) ▷ update score based on feedback 12: iteration ← iteration + 1 13: return d algorithm 4 summarization-based refinement require: diagram d, source pdf p ensure: refined diagram d ref 1: iteration ← 0, maxiterations ← 3, threshold ← 4.5, score ← 0 2: while @xmath10: c score , c f eedback ← c critic (d, p ) 4: f score , f f eedback ← f critic (d, p ) 5: l score , l f eedback ← l critic (d) 6: initialize allf eedback ← {} 7: if score(c score ) < threshold then 8: allf eedback ← allf eedback ∪ {c f eedback } 9: if score(f score ) < threshold then 10: allf eedback ← allf eedback ∪ {f f eedback } 11: if score(l score ) < threshold then 12: allf eedback ← allf eedback ∪ {l f eedback } 13: d ← ref inediagram(d, allf eedback) 14: score ← min(c score , f score , l score ) 15: iteration ← iteration + 1 16: d ref ← d 17: return d ref