U ni T ran S e R : A Unified Transformer Semantic Representation Framework for Multimodal Task-Oriented Dialog System (2022)

Abstract: As a more natural and intelligent interaction manner, multimodal task-oriented dialog system recently has received great attention and many remarkable progresses have been achieved. Nevertheless, almost all existing studies follow the pipeline to first learn intra-modal features separately and then conduct simple feature concatenation or attention-based feature fusion to generate responses, which hampers them from learning inter-modal interactions and conducting cross-modal feature alignment for generating more intention-aware responses. To address these issues, we propose UniTranSeR, a Unified Transformer Semantic Representation framework with feature alignment and intention reasoning for multimodal dialog systems. Specifically, we first embed the multimodal features into a unified Transformer semantic space to prompt inter-modal interactions, and then devise a feature alignment and intention reasoning (FAIR) layer to perform cross-modal entity alignment and fine-grained key-value reasoning, so as to effectively identify user’s intention for generating more accurate responses. Experimental results verify the effectiveness of UniTranSeR, showing that it significantly outperforms state-of-the-art approaches on the representative MMD dataset.

## introduction the multimodal task-oriented dialog systems are designed to help users achieve specific goals such as clothing recommendation or restaurant reservation, which is in growing demand in the current business environment. as a leading study, @xcite released a multimodal dialog dataset (mmd) in the online retail domain. based on such a benchmark dataset, many multimodal dialog models incorporating domain knowledge have recently been proposed @xcite ., name jacket material leather color black image ... ... jacket jeans sling bag necklace platform heels passion for jacket celebrities celebrities histogram style tips graph attributes table ## methodology the proposed unitranser mainly comprises three parts: unified-modal transformer semantic (uts) encoder ( sec. 3.1 the uts encoder is used to project all the multimodal features into a unified vector space for inter-modal interactions, while the fair layer is designed to align cross-modal hidden features, with textual features and visual features from previous uts encoder as inputs. similar to magic @xcite , our htr decoder is designed to decode three types of responses: general responses that refer to the highly frequent responses (e.g., courtesy greetings) in the conversation, such as "how can i help you?"; intention-aware responses that refer to the task-oriented utterances, such as "found some similar black leather-jackets for you"; and multimodal responses that refer to the intentionaware responses with image output. the response type is determined by a query vector q from the fair layer, in which an intention classifier is trained to decide which kind of response should be given out. ## conclusion in this paper, we propose a unified transformer semantic representation framework with feature alignment and intention reasoning, referred to uni-transer. specifically, we project the multimodal features into a unified semantic space by utilizing a transformer encoder to prompt inter-modal interactions. we further design a feature alignment and intention reasoning layer to conduct cross-modal feature alignment and fine-grained intention rea-soning, with the objective of generating more accurate and intention-aware responses. experiments on the representative mmd dataset demonstrate the effectiveness and superior performance of our unitranser model in both automatic and human evaluation.