M c Q ueen: a Benchmark for Multimodal Conversational Query Rewrite (2022)

Abstract: The task of query rewrite aims to convert an in-context query to its fully-specified version where ellipsis and coreference are completed and referred-back according to the history context. Although much progress has been made, less efforts have been paid to real scenario conversations that involve drawing information from more than one modalities. In this paper, we propose the task of multimodal conversational query rewrite (McQR), which performs query rewrite under the multimodal visual conversation setting. We collect a large-scale dataset named McQueen based on manual annotation, which contains 15k visual conversations and over 80k queries where each one is associated with a fully-specified rewrite version. In addition, for entities appearing in the rewrite, we provide the corresponding image box annotation. We then use the McQueen dataset to benchmark a state-of-the-art method for effectively tackling the McQR task, which is based on a multimodal pre-trained model with pointer generator. Extensive experiments are performed to demonstrate the effectiveness of our model on this task.

## introduction recent years have witnessed an increasing attention in conversational-related tasks, such as conversational question answering @xcite , visual conversation modeling @xcite , etc. one main challenge in multi-turn conversation modeling is that information from context history is easy to be abbreviated or omitted in the follow-up queries, causing the so-called coreference and ellipsis. to address this concern, the task of query rewrite @xcite @xcite aims to reconstruct the original query to a fully specified form based on its history context. the rewrite eliminates the coreference and ellipsis in the original query without changing its semantic information, thus helping turn the more challenging multi-turn conversation modeling problem to a single-turn version. following this line, several attempts have been made in the query rewrite task which achieve decent performance on the natural language level. nevertheless, conversations in real scenario tend to involve knowledge from more than one modalities, such as vision, text, speech, etc. information from different modalities is not handled in isolation, but often integrated together to improve the quality of perception and understanding. for example, as shown in figure 1 , in the first turn of the visual conversation, with the lack of context, the user directly uses the pronoun "it" to refer to the dog in the image. in the third turn, for ellipsis that does not appear in the context history, in order to perform ellipsis completion, one also needs to find clues from the corresponding image. rewriting the query under the circumstance where grounding outside the text information is needed poses more challenges to traditional query rewrite models that based only on textual features. in this paper, we propose the task of multimodal conversational query rewrite (mcqr), which aims to perform query rewrite under the multimodal visual conversation setting. to achieve this goal, we collect a large-scale dataset called mcqueen. specifically, for each visual conversation consisting of an image and the corresponding history questionanswer context, we provide manual rewrite for the query, where the coreference resolution and ellipsis completion are performed respectively. furthermore, in order to assist downstream tasks such as coreference entity detection, for all the entities appearing in the rewrite, we annotate the image boxes for representing their corresponding image area. we then use the mcqueen dataset to benchmark a state-of-the-art method for effectively tackling the mcqr task. inspired by the big success of pre-trained models such as bert @xcite , our model is based on a multimodal pretrained model where interactions between different modalities can be better captured. furthermore, we enhance the model with a pointer generator specially designed for the multimodal transformer blocks @xcite , so that the rewritten query is either generated from scratch or copied from certain contextual parts with high attention weights. extensive experiments are conducted to compare our method with several state-of-the-art methods. our model outperforms all the methods in both the mcqr task and two subtasks. we further perform analysis to investigate the role of different modalities in this task and demonstrate that the introduction of image information provides extra guidance for the concerned query rewrite task. in summary, the contribution of our paper lies in three folds: ## error case analysis as shown in figure 6 , we demonstrate the most common error cases of our model, which can be classified into several cases including bad sentence structure, wrong coreference result, inaccurate omission completion, etc. according to the figure, we can see that pre-trained language models can generate sentences that have a proper format, but still have some problems in understanding the deep semantic correlation in the visual conversations. for example, in the top two cases, although the coreference seems simple for human readers, it poses challenges for machines other than directly making a copy from context. specifically, for complicated cases such as when coreference and ellipsis appear at the same time, when the conversation is long, and when the omitted information requires reasoning between history entities (e.g. the bottom cases), there still remains much space to explore. ## conclusion and future work we propose the task of multimodal conversational query rewrite (mcqr), which aims to perform query rewrite under a multi-turn visual conversation. to facilitate the research, we benchmark a large-scale dataset with manual annotation which covers 15k visual conversations with more than 80k rewrites. we also provide image box annotation of entities appearing in the rewrites. accordingly, we propose a model based on an existing multimodal pre-trained model and further enhance it with a pointer generator. extensive experiments are performed to show the effectiveness of our model.