FACTUAL : A Benchmark for Faithful and Consistent Textual Scene Graph Parsing (2023)

Abstract: Textual scene graph parsing has become increasingly important in various vision-language applications, including image caption evaluation and image retrieval. However, existing scene graph parsers that convert image captions into scene graphs often suffer from two types of errors. First, the generated scene graphs fail to capture the true semantics of the captions or the corresponding images, resulting in a lack of faithfulness. Second, the generated scene graphs have high inconsistency, with the same semantics represented by different annotations. To address these challenges, we propose a novel dataset, which involves re-annotating the captions in Visual Genome (VG) using a new intermediate representation called FACTUAL-MR. FACTUAL-MR can be directly converted into faithful and consistent scene graph annotations. Our experimental results clearly demonstrate that the parser trained on our dataset outperforms existing approaches in terms of faithfulness and consistency. This improvement leads to a significant performance boost in both image caption evaluation and zero-shot image retrieval tasks. Furthermore, we introduce a novel metric for measuring scene graph similarity, which, when combined with the improved scene graph parser, achieves state-of-the-art (SOTA) results on multiple benchmark datasets for the aforementioned tasks.

## introduction a scene graph is a representation that describes the contents of a visual scene, including objects, their attributes, and the relationships between them. the grounding of a scene graph with an image or a text can provide significant benefits for various vision-language tasks, such as image caption evaluation @xcite and image re-* the two authors contributed equally to this work. trieval @xcite . therefore, transduction of image descriptions into scene graphs through textual scene graph parsing has been a crucial vision-language research area. accurately generating scene graphs that capture intersected information from images and their corresponding descriptions is crucial for a successful textual parser. however, current baseline parsers often generate unfaithful scene graphs that fail to represent the complete intersected information or generate semantically correct graphs, as shown in figure 1 . furthermore, inconsistencies exist in the outputs of scene graph parsers, as depicted in the same figure, where "tennis" is interpreted as an attribute in one graph and as a part of an object in another graph. such inconsistencies can severely impact downstream tasks of textual scene graph parsers, especially when they produce different graphs for a semantic unit, such as a phrase, across various captions, despite they carry the same semantic meaning. upon inspection, we hypothesize that the issues of unfaithfulness and inconsistency arise due to the inherent shortcomings of scene graph parsing algorithms and limitations within the datasets. one widely utilized parser, spice-parser @xcite , is known for converting caption dependency graphs into scene graphs using predefined rules, which can result in error propagation. furthermore, the dependency graphs may not adequately capture the semantic characteristics of scene graphs, as dependency graphs primarily focus on syntactical relationships. additionally, the limitations of the datasets contribute to the problems as well. as demonstrated in figure 1 , the largest scene graph dataset, vg @xcite , includes notable annotation issues regarding faithfulness and inconsistency. to address the aforementioned issues, we create a high-quality scene graph dataset for training parsers. we firmly believe that the problems of unfaithfulness and inconsistency within the dataset can be effectively resolved by incorporating two key measures: i) employing rigorous definitions for the literals and ii) implementing strict quality control during the annotation process. therefore, we propose a novel intermediate meaning representation (mr) coined as factual-mr, which ensures faithful and consistent textual scene graph parsing. factual-mr is a semantic representation that can be deterministically mapped to the scene graph, thereby avoiding the issues that arise from converting syntactical graphs into scene graphs. the annotation of factual-mrs can be divided into manageable sub-tasks, allowing us to easily control the quality of annotations in each sub-task and ensure their faithfulness. furthermore, the literals within the factual-mrs are precisely defined to ensure consistency in textual scene graph parsing annotations. as a result, we re-annotate captions sampled from the vg dataset with factual-mrs, enabling us to leverage the existing scene graph annotations from vg. additionally, in order to further enhance the advantages provided by the scene graph parsing for its downstream tasks, we propose a simple yet effective metric called softspice. this metric calculates graph similarity and significantly improves the performance of vision-language tasks that leverage scene graphs. overall, the key contributions are as follows: ## related work grounding a scene graph with an image or image description can be beneficial for a variety of downstream tasks, such as image retrieval @xcite , image caption evaluation @xcite and image captioning @xcite . currently, there are three main research directions to scene graph parsing: those that focus on parsing images @xcite @xcite @xcite ), text @xcite @xcite @xcite , or both modalities @xcite into scene graphs. parsing images involves utilizing an object detection model to identify the location and class of objects, as well as classifiers to determine the relationships and attributes of the objects. textual scene graph parsing employs techniques such as the sequence-to-sequence model @xcite to parse image descriptions into linearized scene graphs @xcite or generate intermediate representations, such as dependency graphs or abstract meaning representation (amr) @xcite , which are then mapped into scene graphs using deterministic rules or machine learning models. however, directly utilizing intermediate representations like dependency graphs or amr often leads to subpar performance in downstream tasks, as emphasized by @xcite , and may even be infeasible for multi-modal tasks requiring annotations for both modalities, given that the intermediate representations only annotate the text. recent studies in parsing both modalities @xcite have primarily utilized textual parsing models to enhance the performance of visual scene graph parsing. our work primarily focuses on textual scene graph parsing. ## textual scene graph parsing a scene graph, as introduced by @xcite , is a formal representation of the objects, their attributes, and the relationships between objects in a visual scene. given a set of object classes c, a set of attribute types a, and a set of predicate types r, a scene graph g is defined as a tuple (o, e), where @xmath0, ..., o n } is a set of objects and @xmath1. each object o @xmath2, a i } is associated with an object class c @xmath3. as depicted in figure 1 , our work linearizes a scene graph into a simplified format. in this format, each fact is represented either as (object, has_attribute, attribute) or as (object sub , p redicate, object obj ), which is consistent with the format of the linearized scene graphs outlined in @xcite ; @xcite . therefore, the textual scene parsing aims to learn a mapping π θ : @xmath4, which translates a textual image description @xmath5. ## experiments we evaluate the effectiveness of our new scene graph benchmark through one intrinsic evaluation and two extrinsic evaluation tasks. ## conclusion we introduce a new intermediate representation, coined factual-mr, which aims to address the issues of faithfulness and consistency for textual scene graph parsers. by utilizing a rigorous annotation process, it is possible to create a large-scale dataset based on factual-mr. our experiments demonstrate that factual-t5, trained on this dataset, is capable of generating consistent scene graphs that are highly faithful to corresponding images and captions. utilizing a novel graph similarity metric, softspice, factual-t5 significantly improve performance in both image caption evaluation and zero-shot image retrieval.