  Recent studies on computer vision mainly focus on natural images that express
real-world scenes. They achieve outstanding performance on diverse tasks such
as visual question answering. Diagram is a special form of visual expression
that frequently appears in the education field and is of great significance for
learners to understand multimodal knowledge. Current research on diagrams
preliminarily focuses on natural disciplines such as Biology and Geography,
whose expressions are still similar to natural images. Another type of diagrams
such as from Computer Science is composed of graphics containing complex
topologies and relations, and research on this type of diagrams is still blank.
The main challenges of graphic diagrams understanding are the rarity of data
and the confusion of semantics, which are mainly reflected in the diversity of
expressions. In this paper, we construct a novel dataset of graphic diagrams
named Computer Science Diagrams (CSDia). It contains more than 1,200 diagrams
and exhaustive annotations of objects and relations. Considering the visual
noises caused by the various expressions in diagrams, we introduce the topology
of diagrams to parse topological structure. After that, we propose Diagram
Parsing Net (DPN) to represent the diagram from three branches: topology,
visual feature, and text, and apply the model to the diagram classification
task to evaluate the ability of diagrams understanding. The results show the
effectiveness of the proposed DPN on diagrams understanding.
