Semi-supervised multimodal coreference resolution in image narrations (2023)

Abstract: In this paper, we study multimodal coreference resolution, specifically where a longer descriptive text, i.e., a narration is paired with an image. This poses significant challenges due to fine-grained image-text alignment, inherent ambiguity present in narrative language, and unavailability of large annotated training sets. To tackle these challenges, we present a data efficient semi-supervised approach that utilizes image-narration pairs to resolve coreferences and narrative grounding in a multimodal context. Our approach incorporates losses for both labeled and unlabeled data within a cross-modal framework. Our evaluation shows that the proposed approach outperforms strong baselines both quantitatively and qualitatively, for the tasks of coreference resolution and narrative grounding.

## introduction in linguistic processing, coreference resolution is a standard task that aims to identify referring expressions such as noun phrases and pronouns that refer to the same entity. it is fundamental to many standard problems including question answering @xcite , sentiment analysis @xcite , summarization @xcite and machine translation @xcite @xcite . in this work, we focus on a multimodal coreference resolution (mcr) scenario where the coreferences occur in a narration paired with an image and also link to an image region as shown in figure 1 . here resolving coreferences is challenging, as mentions referring to different entities can be very similar when encoded by a language model, e.g., one boy, the other boy, the boy. hence it demands a fine-grained understanding of each modality and as well as across them. in particular, it requires simultaneously grounding instances by we can see two small boys are sitting on a white color mat and in that one boy is crying and he is wearing a yellow color t-shirt and grey color short. the other boy is wearing white color t-shirt and cream color short and he is also holding some object in his hand. on the head of the boy we can see a white color sticker and on their t-shirts we can see some text and designs also. figure 1 : example image-narration pair from the coreferenced image narratives dataset @xcite . phrases marked in the same color corefer to the same entity which are also grounded in the image. we do not show singletons for brevity. mcr has recently gained increasing attention, with several notable studies @xcite @xcite @xcite @xcite . however, many of them focus on images with simple short sentences, such as 'a woman is driving a motorcycle. is she wearing a helmet?' @xcite , or are limited to identifying movie characters or people @xcite . more recently, @xcite introduced a challenging and unconstrained mcr problem (see figure 1 ) including a dataset, coreferenced image narratives (cin), with both people and objects as referents with long textual descriptions (narrations). as manually annotating a large dataset with coreferencing and grounding labels is expensive, the authors provide annotations only for evaluation purposes. they also propose a weakly supervised method that learns to jointly ground mentions in images and use them as anchors along with prior linguistic rules @xcite to group coreferring men-tions from only image and narration pairs without the annotations. the method has multiple shortcomings: (1) weakly supervised grounding fails to disambiguate multiple instances of the same object class, boy (one boy, the other boy), (2) language rules such as exact match of phrases are either too strict or too generic, e.g., pronoun match, linking pronouns to one antecedent (one boy, he, he, his) and, (3) they require an additional modality, mouse traces to learn coreferences which can be expensive to obtain. motivated by these limitations, we argue that it difficult to successfully resolve coreferences from only image-narration pairs in cases where multiple instances of the same object category are present; this situation is common coincides with a mention in the narration. since full manual annotations of coreference chains and bounding boxes is expensive, we propose to resolve coreferences and ground mentions in a semi-supervised setting where only a few data samples are labeled. our approach involves a customized multi-modal fusion model that combines image region features and mention features from narrations through crossattention @xcite . we investigate different task-specific losses for training on labeled and unlabeled data, and show that naively combining training on the labeled and pseudo-labeled data suffers from severe overfitting @xcite . hence, we propose a robust loss function and thresholding-based training scheme to effectively learn from the unlabeled set. this novel approach results in consistent performance improvements with the inclusion of unlabeled data during training. our main contributions are (1) a vision-language framework for mcr trained on a small labeled and an unlabeled dataset, (2) novel task-specific losses (on both labeled and pseudo-labeled data) for learning joint multi-modal embeddings for coreference resolution while simultaneously improving narrative grounding, (3) extensive evaluation of our proposed method on the cin dataset and ablation studies to validate our design choices, showing consistent performance gains compared to baselines on coreference resolution and narrative grounding. ## related work multimodal coreference resolution. mcr involves comprehending the contextual information in language and establishing connections with spe-cific regions in an image. recently, considerable efforts have been dedicated to developing datasets that can effectively address this intricate task. @xcite introduced the valse dataset, which encompasses various coreference scenarios. however, this dataset focuses on the downstream task of visual question answering without evaluating coreference resolution or grounding. hence, we evaluate our method on cin dataset @xcite that contains coreference chains and grounding annotations. another approach to mcr datasets involves linking people's names mentioned in the text to corresponding images and resolving pronouns that connect to those specific names @xcite @xcite . however, our main focus is to resolve coreferences in a generic scenario (with visual complexity) unlike the others that are either limited to only people names/characters @xcite @xcite or have simple sentences @xcite . vision-language learning. existing work on vision and language understanding employs either pre-trained object detector features @xcite as an image encoder, vit @xcite or a cnn (simonyan and zisserman, 2014) combined with a transformer-based text encoder @xcite . to model crossmodal interaction between the image and text encoders, uniter @xcite , albef @xcite and vinvl @xcite employ a multimodal encoder. they are pre-trained on large-scale image-caption pairs such as coco @xcite , conceptual captions @xcite , visual genome @xcite . the pre-training objectives are implemented with image-text contrastive loss, masked language modeling, and image-text matching loss. our method is inspired by these architectures and is trained using a set of self-supervised and task-based objectives in a semi-supervised learning fashion. semi-supervised learning. there is a large body of work in semi-supervised learning @xcite @xcite . these methods typically exploit unlabeled data via either pseudo-labeling with small amounts of labeled data @xcite @xcite @xcite or by enforcing consistency regularization @xcite on the unlabeled data to produce consistent predictions over various perturbations of the same input by applying several augmentation strategies @xcite @xcite . our method draws inspiration from pseudo-labeling literature and uses a robust loss function and thresholding to counter overfitting to pseudo-labels. ## experiments datasets. we evaluate our proposed method on the cin dataset @xcite that consists of 1000 test and 880 validation image-narration pairs from the flickr30k split of the localized narratives dataset (pont-tuset et al., 2020) annotated with coreference chains and bounding boxes. we use the test split of the cin dataset to report the performance on cr and narrative grounding. the annotations from the validation split are used as the small labeled set during training. the unlabeled dataset is the flickr30k training subset of the localized narratives dataset, which consists of 50k image-narration pairs but is not annotated with bounding boxes or coreference chains. implementation details. for the image regions, we extract bounding box regions, visual features and object class labels using the faster-rcnn object detector @xcite as in @xcite . we use a 4-layer transformer architecture for the text encoder and the multi-modal encoder similar to the albef @xcite framework. the weights of the transformer encoders are initialized with the first four layers of bert @xcite . the visual encoder is a stack of two transformer encoder layers. each transformer encoder layer includes a multi-head self-attention layer and an ffn. there are two heads in the multi-head attention layer, and two fc layers followed by relu activation layers in the ffn. training details are in the appendix. evaluation. we report results for coreference resolution and narrative grounding. for the former, we use the standard conll f1 score which is the average of three coreference-based metrics: muc, b 3 and ceaf ϕ4 . for the latter, we follow @xcite and report the grounding accuracy for both noun phrases and pronouns. more precisely, if the overlap between the ground-truth box and the predicted box is greater than 0.5, then it is considered to be a correct prediction. ## conclusion in conclusion, this paper addresses the challenging task of multimodal coreference resolution where an image is accompanied by a longer descriptive text. we propose a data efficient semi-supervised approach that incorporates task-based losses for both labeled and unlabeled data, operating within a cross-modal framework. our method achieves remarkable results for cr and narrative grounding tasks on the cin dataset, showcasing its effectiveness in handling the complexities of mcr. in the future, we plan to investigate how the power of pre-training combined with semi-supervised finetuning can be fully utilized for the task of mcr.