GRAVL - BERT : Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution (2022)

Abstract: Learning from multimodal data has become a popular research topic in recent years. Multimodal coreference resolution (MCR) is an important task in this area. MCR involves resolving the references across different modalities, e.g., text and images, which is a crucial capability for building next-generation conversational agents. MCR is challenging as it requires encoding information from different modalities and modeling associations between them. Although significant progress has been made for visual-linguistic tasks such as visual grounding, most of the current works involve single turn utterances and focus on simple coreference resolutions. In this work, we propose an MCR model that resolves coreferences made in multi-turn dialogues with scene images. We present GRAVL-BERT, a unified MCR framework which combines visual relationships between objects, background scenes, dialogue, and metadata by integrating Graph Neural Networks with VL-BERT. We present results on the SIMMC 2.0 multimodal conversational dataset, achieving the rank-1 on the DSTC-10 SIMMC 2.0 MCR challenge with F1 score 0.783. Our code is available at https://github.com/alexa/gravl-bert .

## introduction powered by advances in machine learning, intelligent agent systems have seen their capacity expand in recent years. devices with intelligent assistants have become ubiquitous in everyday life. these systems can handle short task-oriented dialogues, but are limited to speech or text inputs and outputs. motivated by the widespread adoption of such agents and multimodal devices with screens that house them, multimodal (visual-linguistic) understanding has become a promising discipline for researchers. the next-generation of intelligent as- * corresponding author: lyleguo@g.ucla.edu † corresponding author: agsanchi@amazon.com figure 1 : an example of multimodal coreference resolution (mcr). based on the image and dialogue context, there can be multiple ways to refer to the highlighted red coat. each of them require different information to locate the target. within multimodal understanding, an important area of research is multimodal coreference resolution (mcr). it is a crucial capability as user references can span across modalities in a multimodal environment. moreover, mcr is a challenging problem even when compared to text coreference resolution and visual question answering tasks because, in mcr, two participants can simultaneously refer to objects while looking at a scene from a shared perspective. in this dynamic frame of reference, the notion of left/right, first/second is constantly shifting and the model cannot rely on the fixed position of objects in the scene. different from both vqa and textual anaphora resolution tasks, there are distinct ways to refer to an object in the mcr task. as the scene may contain a large number of similar objects, it is natural to refer to them by relative position with respect to other objects, front and back w.r.t the camera, and w.r.t. accessory objects like shelves and tables, so as to easily indicate the target object(s). such references are unusual in other related tasks. furthermore, as our focus is on the mcr task within task-oriented dialogues, the objects being referenced are typically associated with a back-end database that also provides metadata information such as price, brand, and size for these objects. this adds an additional dimension to the coreferencing task as the users can also refer to objects based on these non-visual metadata attributes. finally, as conversations involve multi-turn dialogue, mcr also requires reasoning over dialogue context to resolve references like "show coat next to the shirt you suggested previously". an example is shown in figure 1 . resolving these coreference cases requires various multimodal information extraction and reasoning capabilities. for example, to identify "the red coat next to the black one", the model must infer the target's visual features (red), as well as its neighborhood information (a black coat). the user might also say "i'll take the first one", if they are choosing from one of the coats offered previously. notable progress has been made on multimodal frameworks. recent models have shown excellent performance on various multimodal tasks including visual question answering (vqa) @xcite , visual commonsense reasoning @xcite , visual grounding @xcite and image captioning @xcite . all these models take both images and text as inputs. both visual and linguistic tokens are sent to an autoencoder to learn shared representation and perform downstream tasks. however, such methods are still not developed in a way to be able to solve complex mcr scenarios. for example, they may fail when a query has dialogue context instead of a single short sentence, or when there are anaphoric references in the query. finally, most of these models are not designed to handle external knowledge sources or to scale to hundreds of objects in a single scene. motivated by these challenges, we propose a new framework, gravl-bert(graphical visual-linguistic bert), that can simultaneously reason over dialogues, objects and their relationships, scene information, and object metadata. our major contributions are as follows: we present results of gravl-bert on the simmc 2.0 dataset @xcite which involves dialogues between a customer and an agent in the shopping domain. we participate in the simmc 2.0 challenge for the task of mcr, where the goal is to resolve the references and identify the target object(s) in the scenes. we achieve sota performance with 0.76 object-level f1 score on devtest set and 0.78 on test set. we note that this dataset is available for research and non-commercial use. visual-linguistic frameworks. multimodal frameworks that support visual (i.e., image, video) and linguistic inputs (i.e., caption, dialogue) can be fine-tuned for various tasks including mcr. in early works, most models encoded visual and linguistic features separately and combined them only at a later stage. for example, both mtn @xcite and lxmert @xcite have two separate encoders for visual and linguistic inputs and then use a query-aware encoder and cross-modality encoder respectively to extract visual features related to query text. in recent works, increasing number of high-performing models adopt an early-fusion strategy. they first extract the regions of interest (roi) features from visual inputs, convert them into token embeddings, and concatenate with text embeddings. then they employ bert @xcite to learn the cross-modal associations and perform different tasks. for instance, vil-bert @xcite ) concatenates text embeddings and image embeddings together and sends them to bert. oscar @xcite , vinvl @xcite , 12-in-1 @xcite , unicoder-vl @xcite , and unified vlp @xcite all improve upon vil-bert by adding better image features, new pretraining strategies, or multiple datasets for pretraining. vl-bert @xcite is currently one of the most popular benchmark frameworks for visual-linguistic tasks. it uses fast r-cnn to extract visual features of objects and scene images, and concatenates them with linguistic token embeddings. the combined features are then fed into a bert module, which is fine-tuned for various downstream tasks. ## methodology problem formulation. given inputs (d, i, n, m, q) where @xmath0, d 2 , ..., d k } is the dialogue text split into k turns, @xmath1, i 2 , ..., i k } are the scene images for the dialogue turns, n @xmath2, n j2 , ...} is the set of objects inside each scene i @xmath3, m @xmath4, m 2 , ...} are the metadata attributes of each object n ij and q is the user query referring to one or more objects, our task is to predict a label y @xmath5, 1} for each object @xmath6. the user query can involve spatial references, visual references, metadata based references, or any combination of these. model details. our model builds upon vl-bert. we extend the framework from single utterance input to multi-turn dialogue input, with each dialogue turn associated with its respective scene image. in addition, the scene objects can have external knowledge base (e.g., metadata) associated with them. the architecture for our model is shown in figure 2 . the model takes 4 different streams of input: linguistic, visual, segment, and position. input from different streams are combined via feature-wise addition. the visual stream consists of the visual features from the whole scene, the candidate object and its surroundings. for each of the visual component, we use fast-rcnn @xcite to extract the features and then augment it with bounding box location to add spatial information. we further add a gcn layer to explicitly capture the relative position of each object with respect to others. the linguistic stream includes the dialogue context, user query and candidate object's metadata. we flatten the structure of metadata and convert it into a string of the form "key1 value1 key2 value2 ...". we also add two special tokens, an integer feature t which denotes the distance from the turn when the object was last offered by agent, and a string s, which indicates that the corresponding visual features (in the visual stream) are from the object's surroundings and not the object itself. the segment stream is used to distinguish the dialogue context, user query and object metadata inputs. it has three different values, corresponding to these input types. the position stream contains token positional embeddings, which is same as the one used in the original bert model. note that, we feed only one object into the model at a time. this allows our system to scale well for scenes that may contain large number of objects. for instance, in the simmc 2.0 dataset, many scenes contain more than 100 objects with metadata sequence length larger than 50 for each object, making it impractical to feed all 5k object instances into the system. we supplement our model with gcn based structure to mitigate strong independence assumptions price $12.99, rating 4.7, . ## conclusion and discussion in this work, we proposed a multimodal framework gravl-bert for mcr task. our contribution lies in systematically combining relevant techniques such as utilizing external knowledge sources (metadata, generated captions), gcn, sampling object's neighborhood, and dialogue-oriented pre-training using a simple bert-based architecture to perform mcr within dialogues grounded in scenes. we improved over the gpt-2 @xcite based baseline by 33.9% absolute and 2.5% over other concurrent work. for future research in the topic we suggest several avenues. as mentioned in section 4.5, to identify objects with complex references, global information is required e.g. number of cabinets in the scene, price ranking among objects. currently, our model is unable to handle this complexity. we believe adding specific encoders to extract these (a) an example with correct predictions. there are two target objects (highlighted with boxes), both of which our model gets right. coreferencing requires both visual ("brown one", "red and white sweaters") and spatial understanding ("middle of the top row", "besides the bright blue jacket"). in the pre-training stage, we trained our model on dialogues using mlm. we expect that applying prompt in this stage may have a promising performance. instead of training the model to learn to predict the randomly masked words, using a carefully designed prompt can teach model to focus on the information that is helpful to our main task. lastly, there are cases where the user provides very general descriptions and the information to resolve the coreference is insufficient. for example, the user refers to "the red sweater" while there are multiple red sweaters in the scene. in this situation, instead of trying to resolve the coreference, the system may attempt to disambiguate. (e.g., "which sweater do you mean?") we expect future work to distinguish these kinds of situations.