CRAPES :Cross-modal Annotation Projection for Visual Semantic Role Labeling (2023)

Abstract: Automatic image comprehension is an important yet challenging task that includes identifying actions in an image and corresponding action participants. Most current approaches to this task, now termed Grounded Situation Recognition (GSR), start by predicting a verb that describes the action and then predict the nouns that can participate in the action as arguments to the verb. This problem formulation limits each image to a single action even though several actions could be depicted. In contrast, text-based Semantic Role Labeling (SRL) aims to label all actions in a sentence, typically resulting in at least two or three predicate argument structures per sentence. We hypothesize that expanding GSR to follow the more liberal SRL text-based approach to action and participant identification could improve image comprehension results. To test this hypothesis and to preserve generalization capabilities, we use general-purpose vision and language components as a front-end. This paper presents our results, a substantial 28.6 point jump in performance on the SWiG dataset, which confirm our hypothesis. We also discuss the benefits of loosely coupled broad-coverage off-the-shelf components which generalized well to out of domain images, and can decrease the need for manual image semantic role annotation.

## introduction automatic image comprehension can positively contribute to many modern applications, such as description generation, cross-modal retrieval, and human-robot interaction. to comprehend an image it is important to identify the action(s) and participants in the action such asan agent (who is performing the action), a patient (who is being affected by the action), and an instrument. to address this problem @xcite proposed the task of grounded situation recognition (gsr). many approaches @xcite @xcite have been proposed to perform the task of gsr. most of these frameworks have two steps: in the first step verbs are predicted, and in the second step nouns and roles are predicted in an auto-regressive manner. some other methods deployed include another layer to refine the quality of detection @xcite @xcite . one fundamental limitation of these models derives from the problem formulation. in the current formulation, verb frames would compete for an image, limiting the expressiveness of the image's semantic representation. in reality, various actions can co-exist in an image, even sharing participants. this limitation of one frame per image is imposed by the predominant dataset of gsr: the swig dataset @xcite . for example, fig-ure 1a depicts a ground-truth (gt) annotation of an image from swig and has a gt annotation only with respect to a drinking frame. in fact, there are other frames, such as holding, wearing. semantic role labeling (srl) of natural text, on the other hand, is a well researched problem in the domain of computational linguistics. semantic role annotation, based on paradigms such as prop-bank or framenet @xcite , is used to train semantic parsers that then convey knowledge about who is doing what to whom, when as predicate-argument structure labeling. in other words, given an action in a sentence, it identifies who is performing the action (the agent), who is affected by the action (the patient), what instrument is being used, etc. to comprehend the meaning of the sentence. semantic roles of a sentence have the capability representing more than one predicate-argument structure for that sentence. current text-based srl systems have gained remarkable accuracy. however, srl of images has yet to enjoy similar success. we hypothesize that expanding gsr to follow the more liberal text-based srl approach to action, participant identification could improve image comprehension results. here, we propose a framework (crapes) with cross-modal annotation projection (ap) for visual semantic role labeling. ap is a well-known paradigm in text-based cross-lingual semantic role labeling @xcite @xcite that has not been previously extended to cross-modal applications. moreover, to preserve generalization capabilities, we focus on reusing general-purpose vision and language (v+l) components and text-based srl components. this framework offers the following advantages over traditional gsr approaches: the idea of grounding nouns in the image was coined by @xcite , thereby proposing the task of gsr and the swig dataset. a recurrent framework with resnet-50 embedding was used to detect the verb and then the noun for each role. a retinanet backbone was used for object grounding. @xcite model visual srl as query based vision reasoning. @xcite adopt a top-down attention model @xcite and deploy inter-dependent queries to model relations among semantic roles. @xcite use a transformer encoder to classify verbs and to create image representations. then the image representation was queried with the concatenation of roles and verbs. however, most of these aforementioned approaches use two-stage frameworks where in the first step the verb is predicted independently and then nouns and roles are predicted in an autoregressive manner depending on the verb. however, subsequent work @xcite identified that this emphasis on the detection of the verb may confuse the prediction. furthermore, verb miss-classification may result in miss-recognition of semantic roles. therefore, they adopted a three-stage framework. in the first two stages candidate verbs and nouns were detected. the third stage mostly refined the prediction. during the detection of the candidate, information flows either from verb to noun @xcite or from noun to verb @xcite . this ignores the semantic dependency in the other direction. moreover, this refinement can be done in only one iteration. @xcite solved these issues by designing an iterative method through message passing between verb and noun prediction modules. recently, @xcite addressed the task of gsr, even though their main goal was to propose a pre-training schema using event based cross-modal alignment. all of these methods are limited to predicting one verb per image. none of these models acknowledge the existence of multiple actions and therefore multiple verb frames. ## approach to detect semantic roles in images we adopted the idea of ap, as discussed above, from cross-lingual semantic role labeling in the text domain. in ap, auto-predicted semantic roles from source language is transferred to a target language using soft word alignments. alignment is learned using large-scale parallel corpus. in the case of gsr we consider the image as our target domain. ## future work current gsr models cannot go beyond the swig dataset. moreover predicted semantic roles are restricted to follow a particular paradigm of srl. on the contrary, having independent v+l enables crapes to work on other image datasets. in addition, having a separate srl module enables extension to other srl paradigms. we performed preliminary experiments on the flickr30k dataset with propbank @xcite annotation. figure 7 depicts one such example. we would like to extend our experiments to the version of flickr30k used in @xcite . however, our preliminary experiments suggest that experiments with flickr30k are more challenging for several reasons. our formulation of crapes can accommodate propbank srl experiments on flickr30k. however, a more rigorous study with human evaluation is required to correctly measure the potential of crapes. therefore, this a critical future direction for us. it requires a new dataset with images annotated with more than one frame. one choice is to extend the swig dataset to accommodate more than one frame per image. another choice is to enhance the current flickr30k annotation. ideally we would do both. however, the current proposed evaluation metrics for gsr are incompatible wih a multi-frame scenario. more robust and appropriate evaluation metrics also need to be developed. ## conclusion in this paper we identified a fundamental issue in the problem formulation of the gsr task. the current formulation limits an image to a single verb frame. we propose an alternate formulation allowing for multiple actions as implemented in crossmodal annotation projection for visual semantic role labeling (crapes). a v+l model trained on image-text parallel corpora and an srl module trained independently on text corpora allow the model to integrate domain-specific knowledge with out-of-domain knowledge, which dramatically improves over the sota by 28.6 points. in addition, crapes can accommodate co-existing verb frames for an image (crapes 1 ) yet can also be trained to select only one verb frame for a given image (crapes 2 ). moreover, inter module independence allows crapes to extend its labeling to alternative paradigms of srl (such as framenet or propbank). however one major area for improvement is grnd-all, that requires better semantic comprehension and guidance of attention weights produced by the v+l module. therefore, improving on grnd-all along with flickr30k and propbank will be our next endeavour. we will also explore extending datasets to have multiple ground truth frames per image and more appropriate evaluation metrics for reporting results on those datasets.