Block Diagram-to-Text: Understanding Block Diagram Images by Generating Natural Language Descriptors (2022)

Abstract: Block diagrams are very popular for representing a workflow or process of a model. Understanding block diagrams by generating summaries can be extremely useful in document summarization. It can also assist people in inferring key insights from block diagrams without requiring a lot of perceptual and cognitive effort. In this paper, we propose a novel task of converting block diagram images into text by presenting a framework called “BloSum”. This framework extracts the contextual meaning from the images in the form of triplets that help the language model in summary generation. We also introduce a new dataset for complex computerized block diagrams, explain the dataset preparation process, and later analyze it. Additionally, to showcase the generalization of the model, we test our method with publicly available handwritten block diagram datasets. Our evaluation with different metrics demonstrates the effectiveness of our approach that outperforms other methods and techniques.

## introduction block diagrams are commonly used to represent a process or workflow of a system, especially the diagrams with different shapes connected with arrows. these types of diagrams are generally found in industry reports, scientific magazines or papers. however, different people use different shapes for a particular notation which makes it quite challenging to understand @xcite . block diagram summarization is a task where the goal is to extract the contextual information and relationship between different shapes or nodes from the image, and summarizes the key points in natural language. there are several key benefits and applications of block diagram summarization. first, most of the documents not only contain text but also block diagrams. in order to summarize a document automatically, artificial intelligence (ai) needs to understand those block diagrams as well. automatic generation of description from a block diagram image will lead to better analysis of the related document. second, descriptive text of a block diagram can be further used for the question and answering (q&a) task @xcite . third, block diagram summaries can assist individuals to recognize important insights from diagrams that they may have missed otherwise. it is a well-known fact that captions or small descriptions help readers to find important keypoints from the diagrams. it can also help writers to compose effective reports and articles on data facts suggested by automatic explanatory texts. block diagram summarization offers one more significant advantage of making diagrams more accessible to visually impaired people. with the help of descriptions, they can read using screen readers and understand what is being presented in the block diagram. regardless of its various advantages and applications, the block diagram summarization problem has not received much attention in the nlp community. we found no literature regarding block diagram summarization. early approaches focus mainly only on the detection of different shapes in the diagram @xcite or converting the handwritten block diagrams to computerized or electronic format @xcite @xcite . but none of them consider about relating text phrases with shapes and arrows which plays an important role in summarization tasks. recently, researchers considered data-driven neural models for describing tabular data @xcite . also, few researchers considered chart-to-text for describing different types of chart images @xcite . however, compared to tables and charts, block diagram serves a different problem which consists of lots of variations and complexity. for example, some diagrams contain a single parent and child node whereas some diagrams contain two or more parents or child nodes with different varieties of arrow structures that makes it more complex. there are two main difficulties in addressing the block diagram summarization task. first, the lack of computerized block diagram dataset makes it difficult to solve the task using deep learning models. to our knowledge, there is no dataset available for computerized block diagrams that contain human written summaries. second, there are no strong baselines for the block diagram summarization task. in this paper, we present a framework called "blosum" that converts the block diagram images into text. this framework extracts the contextual meaning and relationships between nodes from the images in the form of @xmath0, relation, @xmath1. triplets play an important role in data-to-text generation @xcite , generally used to represent knowledge graph (kg) @xcite . additionally, we present a new dataset for computerized block diagrams (cbd) consisting of 502 diagrams with more than 13,000 annotated elements (shapes, edges, and text phrases) and make our dataset available on github foot_0 . we introduce three variations of problems mainly based on arrow structure: (i) break arrows (that have some gap in between an arrow) (ii) connected arrows (where two or more arrows are interlinked together) (iii) normal arrows (single arrows including both thin and thick types). these different scenarios motivate us to combine computer vision (cv) and natural language generation (nlg) techniques. additionally, we test the blosum with publicly available handwritten block diagram datasets i.e., fc_a @xcite and fc_b @xcite to demonstrate the generalization of the model. for a fair comparison, we extend those two datasets by writing high-quality summaries and triplets. the main contributions of this paper are as follows; image to data generation earlier, julca-aguilar and hirata (2018) trained the well-known faster r-cnn object detection pipeline. standard object-based approaches are unable to identify edges because the arrow bounding boxes are insufficient to identify the relationship between shapes and arrows. to overcome this limitation, schäfer et al. (2021) added an arrow keypoint predictor to faster r-cnn. this keypoint predictor predicted the head and tail keypoints of an arrow that helped in finding the relationship between shapes. however, the major downside of this work is that they failed to detect and relate text phrases with shapes and arrows. moreover, schäfer and stuckenschmidt (2021) outperformed the arrow r-cnn by modeling arrow as a relation between two shapes, and not as standalone objects with bounding boxes. they improved the performance in detecting arrows, but again didn't consider about text phrases relationship with shapes. our work addresses these issues by considering the text phrase relations for both simple and complex diagrams. balaji et al. (2018) proposes chart summarization based on a predefined template. a key limitation of template-based work is their limited scalability and flexibility. moreover, they offer little variation with regard to grammatical styles and lexical choices. in contrast, we focus on the generic block diagram-totext problem without using any predefined template that contains lots of variations and complexity. data to text generation data to text model aims to generate a descriptive text from data or a set of triplets. the task of generating text from data started after the creation of sports summaries from game records @xcite . recent efforts made use of neural encoderdecoder mechanisms @xcite @xcite . pre-trained language models (plms) such as bert @xcite , xlnet @xcite , or roberta @xcite have established a baseline performance for many natural language understanding (nlu) tasks. however, for many nlg tasks, generative plms had set a benchmark such as gpt (brown et al., 2020), bart @xcite , and t5 @xcite . t5 model has also state-of-the-art performance on more than twenty natural language processing (nlp) tasks such as glue @xcite , cnn/daily mail @xcite , superglue @xcite , squad @xcite and many more. it's very uncommon for a single technique to yield consistent advancement across so many tasks. based on this, we adopt the t5 model in our framework for generating sentences. image captioning due to the availability of large-scale datasets, there has been quick advancement in image captioning @xcite . zhang et al. (2021) developed a model to summarize objects from images using an object detection model while sidorov et al. (2020) generate captions from images by extracting a text with the help of ocr. but images with realworld scenes and objects are totally different from block diagrams. real-world scenes don't have a very complex relationship between objects whereas block diagrams contain relationships between different nodes that carry both textual and mathematical information. this makes the block diagram-totext problem different from image captioning. 3 datasets block diagram summarization task uses both object detection and language models, which require a lot of annotated images with high-quality summaries written by humans. to the best of our knowledge, there is no publicly available dataset for computerized block diagrams that satisfies our needs. in this work, we introduce a new dataset cbd for complex computerized block diagrams. we explain all datasets along with the process making of cbd in arrow type split diagrams symbols break train 56 1496 validation 19 528 test 19 451 connected train 64 1694 validation 22 612 test 22 563 normal train 180 4590 validation 65 1806 test 55 1360 ## models in this section, we explain our proposed architecture "blosum" and all other models used for the block diagram summarization task. ## error analysis and challenges to better analyze the results, we manually choose 50 samples from each dataset obtained by different models as shown in figure 6 . this analysis uncovers some key challenges for vision as well as language tasks that we describe below. additional sample outputs are provided in appendix a.5. vision challenge due to improper detection of some shapes and texts, arrow prediction detects some extra or neglects some pre-existing arrows. this results in the wrong prediction of the triplets, which directly affects the language model in summary generation. another vision challenge is related to ocr. block diagrams contain a lot of important information. since ocrs are not 100% accurate, it detects some wrong texts which lead to error in the facts. more accurate data extraction is necessary for block diagrams. imaginary prediction imaginary prediction problem is very common for language models in the data-to-text task. models sometimes predict some imaginary text which is not relevant to the block diagram image. some previous works @xcite face the same problem for the data-to-text task. large scale dataset neural models generally require large-scale datasets. however, our dataset covers a lot of variations but is not big enough. collecting block diagram images, annotations and their human written summaries are difficult tasks as it requires a lot of manual labor. ## conclusion we have presented a novel task of generating textual descriptions from an image of a block diagram. for this purpose, we propose a new architecture called "blosum" that extracts the contextual meaning from the diagram in the form of triplets. additionally, we introduce a new dataset cbd for complex computerized block diagrams with their annotated objects, triplets, and human written summaries. moreover, for showing the generalization of our model, we tested and extended the publicly available handwritten block diagram datasets i.e., fc_a and fc_b by adding triplets and summaries. this extended dataset can also be used for other data-to-text tasks. our evaluation with different metrics shows a promising result and outperforms other methods and also reveals some of the unique challenges for this task.