S imple MTOD : A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation (2023)

Abstract: SimpleMTOD is a simple language model which recasts several sub-tasks in multimodal task-oriented dialogues as sequence prediction tasks. SimpleMTOD is built on a large-scale transformer-based auto-regressive architecture, which has already proven to be successful in uni-modal task-oriented dialogues, and effectively leverages transfer learning from pretrained GPT-2. In-order to capture the semantics of visual scenes, we introduce both local and de-localized tokens for objects within a scene. De-localized tokens represent the type of an object rather than the specific object itself and so possess a consistent meaning across the dataset. SimpleMTOD achieves a state-of-the-art BLEU score (0.327) in the Response Generation sub-task of the SIMMC 2.0 test-std dataset while performing on par in other multimodal sub-tasks: Disambiguation, Coreference Resolution, and Dialog State Tracking. This is despite taking a minimalist approach for extracting visual (and non-visual) informa- tion. In addition the model does not rely on task-specific architectural changes such as classification heads.

## introduction multimodal conversational agents have witnessed a rapidly growing level of interest among the conversational ai community as well as within the computer vision community. most multimodal conversational datasets to-date are an extension of visual question answering (vqa) @xcite . consequently building upon the success of other visio-linguistic tasks such as vqa, state-of-the-art multimodal conversational agents commonly depend on non-autoregressive models @xcite most of which are based on bert @xcite . however, dialogues with such systems significantly differ from what the conversational ai com-munity has typically viewed as a multi-turn dialogue. first, most of the current multimodal dialogue datasets are focused on querying the visual content whereas external knowledge bases have been an integral part of traditional unimodal dialogue datasets @xcite . second, in traditional unimodal dialogues, co-reference resolution (explicitly or implicitly) plays a major role within the dialogues. additionally, state-of-the-art unimodal conversational agents predominantly rely on gpt-based auto-regressive models @xcite due to their proven language generation capabilities @xcite @xcite . the simmc 2.0 @xcite taskoriented dialogue dataset bridges this gap between multimodality and the more traditional view of a multi-turn dialogue. due to the simultaneous presence of signals from multiple modalities, which a user can refer to at any point in the conversation, the multimodal task-oriented dialogues proposed in the simmc 2.0 are challenging compared to both text-only counterparts and image querying dialogue datasets. in spite of the inherent complexity of multimodal dialogues, we propose simplemtod, recasting all sub-tasks into a simple language model. sim-plemtod combines the idea of 'de-localized visual object representations' with a gpt-like autoregressive architecture. the idea of de-localized representations stems from the analogous process of de-lexicalization that has been extensively used in task-oriented dialogues. in de-lexicalization @xcite , slot-values such as vegan are replaced by a more general abstracted token such as food-type. likewise, when de-localized, objects are represented by the catalogue type of the object instance rather than the instance itself. these delocalized tokens then possess a consistent meaning throughout the dataset. the main objective this work is to evaluate the effectiveness of de-localized object representations within simplemtod. despite the simplicity, sim-plemtod achieves the state-of-the-art bleu score of 0.327 for assistant response generation in the simmc2.0 test-std foot_0 dataset . furthermore, the model achieves an accuracy of 93.6% in multimodal disambiguation (mm-disambiguation), object-f1 of 68.1% in multimodal co-reference resolution (mm-coref), and 87.7% (slot-f1) and 95.8 (intent-f1) in multimodal dialogue state tracking (mm-dst). other than the proposed benchmark settings, we also evaluate simplem-tod in an end-to-end setting. major contributions of our work are as follows: ## background traditional task-oriented dialogue datasets consist of a dialogue corpus, a dialogue ontology with a pre-defined set of slot-value pairs, and annotations required for related sub-tasks in a set of domains @xcite . the simmc 2.0 dataset follows a similar structure and contains dialogues in both the fashion and the furniture domains. however, in the simmc 2.0 multimodal dialogue corpus, each dialogue is also associated with an image representing the scene where each dialogue takes place. a scene is made by re-arranging a known set of items (objects) in different configurations. along with the raw-image, the dataset provides a file (scene json) containing details of the images such as objects and relationships between objects. furthermore, a meta-data file contains visual and non-visual attributes of objects that recur within a scene. ## methods in the first part of this section, we model multimodal task oriented dialogues as a sequence generation task. we define the problem in a more general setup and discuss some empirical limitations applied to the model. ## experiments in section 3.1 we defined an end-to-end setting for simplemtod. however, some of the benchmark tasks allow more ground-truth information to be utilized during training and inference time. for the mm-disambiguation task, we consider two setups. in the task-specific scenario, we train the model to predict yes or no tokens directly from context c t . in the end-to-end setup, we consider the label to be yes only if the system intent predicted is to disambiguate. two similar setups are considered for mm-coref as well. it should be noted that end-to-end version of simplemtod predicts de-localized tokens with spatial information and we obtain the canonical object id by reversing the de-localization process explained in section 3.2. if multiple objects were found in the same region with same catalogue-item token, the area of the object bounding box is used as a tie-breaker. in the case of assistant response generation, the benchmark task defined in simmc 2.0 allows groundtruth system belief state to be used as an input. therefore, we evaluate both from action response generation as well as end-to-end setting. ## discussion in order to understand the behaviour of simplem-tod, we use gradient-based salience @xcite provided with the ecco framework @xcite . using ecco, we inspect salience ## conclusion we explore a simple, single generative architecture (simplemtod) for several sub-tasks in multimodal task-oriented dialogues. we build on largescale auto-regressive transformer-based language modeling, which has been effectively utilized in task-oriented dialogues, and formalize the multimodal task-oriented dialogue as a sequence prediction task. our model employs a 'de-localization' mechanism for visual object representation that ensures the consistency of those tokens throughout the dataset. furthermore, we encoded spatial infor-figure 7 : salience score heat-map when predicting the correct intent token request:compare for the dialogue turn with final utterance "can you tell me the brands for the purple and maroon ones on the left and how much they are?" without providing scene information mation of object instances with a very small number of special (globally consistent) tokens. despite the simplicity in representing visual information, our model demonstrates comparable or better performance with models that heavily rely on visual feature extraction, on four multimodal sub-tasks in the simmc2.0 challenge.