Unveiling the Power of Integration: Block Diagram Summarization through Local-Global Fusion (2024)

Abstract: Block Diagrams play an essential role in visualizing the relationships between components or systems. Generating summaries of block diagrams is important for document understanding or question answering (QA) tasks by providing concise overviews of complex systems. However, it’s a challenging task as it requires compressing complex relationships into informative descriptions. In this paper, we present “BlockNet”, a fusion framework that summarizes block diagrams by integrating local and global information, catering to both English and Korean languages. Additionally, we introduce a new multilingual method to produce block diagram data, resulting in a high-quality dataset called “BD-EnKo”. In BlockNet, we develop “BlockSplit”, an Optical Character Recognition (OCR) based algorithm employing the divide-and-conquer principle for local information extraction. We train an OCR-free transformer architecture for global information extraction using BD-EnKo and public data. To assess the effectiveness of our model, we conduct thorough experiments on different datasets. The assessment shows that BlockNet surpasses all previous methods and models, including GPT-4V, for block diagram summarization.

## introduction document artificial intelligence (document ai) is gaining momentum across industries for streamlining document processes, enhancing efficiency, and extracting insights from unstructured data. downstream tasks span various domains and paradigms, such as document qa @xcite , layout detection @xcite , classification @xcite , and more. documents contain not only text, images, charts, or tables but also the most important block diagram, which is often overlooked but plays a crucial role in illustrating systems, processes, or relationships. block diagrams are visual representations of systems, processes, or relationships using blocks or boxes to represent components or elements, and lines or arrows to show the relationships between them. they are particularly useful in technical documentation, engineering reports, business models, scientific papers, and more. block diagrams can vary in complexity from simple representations of basic systems to more complex systems with numerous interconnected components. summaries from block diagrams offer numerous advantages, including reducing manual effort, enhancing productivity, integrating with ai systems for analysis, document ai, qa, and knowledge representation. they improve accessibility for visually impaired individuals, facilitate efficient information retrieval, support collaboration among team members, aid decision-making, and streamline workflows. these summaries also convert complex information into concise descriptions, aiding comprehension and saving time compared to manual analysis. despite its numerous advantages and potential applications, the block diagram summarization problem remains relatively overlooked within the natural language processing (nlp) community. we encountered a scarcity of literature on block diagram summarization. previous approaches primarily focused on detecting different shapes within the diagrams (julca-aguilar and hirata, 2018) or converting handwritten block diagrams into electronic formats @xcite @xcite . however, approaches failed to address the vital task of linking text phrases with shapes and arrows, crucial for effective summarization. to the best of our knowledge, only a few methods have effectively accomplished diagram summarization by combining object detection with pre-trained language models @xcite . neverthe-less, their results are subject to several limitations such as hallucination problem, ocr errors, lack of large dataset, and support limited diagrams. even the latest powerful vision language model, gpt-4v @xcite , struggles with efficiency, particularly in the task of converting block diagrams to text, often lacking detailed information. addressing block diagram summarization encounters two main challenges: the absence of a large dataset and the lack of robust baselines. in this paper, we propose a fusion-based framework "blocknet", comprising three main components: a local information extractor, a global information extractor, and an integrator. the local extractor, based on our algorithm "blocksplit", follows the divide-and-conquer principle to split block diagrams into subunits, each containing two connected nodes or symbols. these subunits are then processed using ocr to extract triplets @xcite . the global extractor utilizes an ocrfree visual document understanding (vdu) transformer model, trained on both english and korean diagram datasets. lastly, the integrator, a large language model (llm), interprets visual information and combines outputs from both extractors. we also introduce a new multilingual data collection approach for block diagram understanding and generation, enabling flexibility and scalability. with this approach, we compile a large, high-quality dataset, "bd-enko", containing 83,394 diagrams with their summaries in english and korean, expandable to any language using our approach. to summarize, our contributions are below: visual document understanding several vdu methods have been developed to interpret semistructured documents and extract vital information, such as receipts and tables @xcite @xcite . early approaches aimed to utilize ocr-independent visual backbones @xcite @xcite but exhibited limited performance. subsequent advancements in ocr @xcite led to the development of various ocr-dependent vdu models by combining with language models @xcite . these methods typically address the task in two stages: 1) extracting text from document images using deep learning-based ocr; 2) comprehending the document as a whole. however, ocrdependent techniques face challenges as ocr errors can negatively impact downstream processes @xcite . to address this, @xcite introduced a transformer-based ocrfree architecture @xcite @xcite but still it faces problems particularly with non-english words. considering these points, we integrate both ocr techniques (local information) and ocr-free models (global information) to address the respective limitations of each. image to data generation earlier, @xcite explored understanding symbolic descriptions in diagrams and suggested a method that links symbolic descriptions with visual attributes to enhance comprehension and analysis. julca-aguilar and hirata (2018) introduced a handwritten block diagram detection system using faster r-cnn. however, their method's inability to recognize edges prevents it from identifying the symbols connected by the arrows, thus limiting its effectiveness in block diagram understanding tasks. arrow r-cnn addresses this by employing an arrow keypoint predictor to learn and predict arrow positions @xcite . furthermore, @xcite improves arrow detection by modeling arrows as relationships between shapes rather than as independent objects with bounding boxes. however, these systems only detect symbol categories without capturing key information conveyed by the block diagram. @xcite introduced the blosum text generator instruction flowchart td a[email] -->|send invoice| b(payment received) b --> c{payment method} c -->|debit card| d[process order] c -->|credit card| e[hold order] c -->|paypal| f[fa:fa-check-circle payment confirmed] e -->|update payment| g[fa:fa-coffee grab a coffee] e -->|cancel order| h[fa:fa-times cancelled order] the email is sent, and if the payment is received, it checks the payment method. if the payment method is debit card, the order is processed. if the payment method is credit card, the order is put on hold. if the payment method is paypal, the payment is confirmed. if the order is on hold, it can be updated by the payment or cancelled. mermaid code +theme, style +instruction block diagram images text descriptions mermaid code generation block diagram text generation block diagram image generation framework, which employs a language model to generate textual descriptions for identified block diagram images, thereby improving tasks such as document summarization. however, language models face challenges such as the illusion problem, where the generated content may lack meaning or reliability compared to the original content. moreover, the dataset that they used for training the model was very small. in contrast, our approach is to integrate the local information (summary of each sub-block) from the diagram as well as the global information (overall summary). moreover, our model is trained on a significantly larger dataset (around 76k) compared to the blosum framework (around 500). ## method in this section, we divide into four subsections. the first explains our unique data generation method. the second discusses the large-scale dataset "bd-enko". the third analyzes the dataset, and fourth presents "blocknet", our fusion-based framework. ## experiments we first describe the experimental setup in section 4.1, followed by a discussion of the metrics employed to measure performance in section 4.2, and at last present the main results in section 4.3. ## ablation study as shown in furthermore, we investigated the feasibility of automating the generation of mermaid code directly from block diagrams extracted from images, with subsequent summarization using gpt-3.5. initially, we trained a transformer encoder-decoder model for this task but encountered considerable challenges. the strict syntax requirements of mermaid code, particularly regarding arrow and text arrangement, posed significant hurdles, as even minor deviations could drastically alter the code's meaning. despite our efforts, the model frequently produced inaccurate outputs due to difficulties in capturing diagram structure accurately. additionally, generating coherent summaries using gpt-3.5 from inaccurately formatted mermaid code proved problematic, leading to errors. ## conclusion in conclusion, this paper introduces the "blocknet" framework, which effectively integrates local and global information from block diagrams to enhance summary generation. our innovative approach to multi-lingual data collection, designed for block diagram understanding and generation, not only provides flexibility and scalability but also facilitates the creation of the diverse and high-quality benchmark dataset "bd-enko" in english and korean languages. through rigorous automatic and human evaluations, our framework showcases stateof-the-art performance on both english and korean diagrams. these findings underscore the significance of emphasizing the challenges that even advanced models like gpt4v face in comprehending block diagrams. moreover, integrating gpt4v as a component in blocknet underscores the crucial role of "local and global" fusion in achieving the superior performances observed. overall, these contributions drive advancements in various fields, including block diagram understanding, document ai, and qa, laying a solid foundation for future research across diverse domains.