Improving Situated Conversational Agents with Step-by-Step Multi-modal Logic Reasoning (2023)

Abstract: To fulfill complex user requirements in a situated conversational scenario, the agent needs to conduct step-by-step multi-modal logic reasoning, which includes locating objects, querying information and searching objects. However, existing methods omit this multi-step procedure and therefore constitutes the risk of shortcuts when making predictions. For example, they may directly copy the information from the dialogue history or simply use the textual description without perform visual reasoning. To address this issue and further boost the system performance, we apply the dual process theory to plug a reasoner into the original transformer based model for step-by-step reasoning. When system 2 completes multi-step reasoning, its output is regarded as final prediction. Our proposed method achieved the 1st rank on the summing scores across all four DSTC-11 SIMMC 2.1 sub-tasks.

## introduction a situated conversational agent @xcite engages in a conversation in an embedded multimodal context, which may involve a virtual environment, a real physical world, etc. this task features language understanding, visual perception, decision feedback, and abundant interactions and reasoning among various modalities. recently, @xcite proposed virtual shopping scenarios based on fashion and furniture, i.e. simmc, and it becomes a popular benchmark for the situated conversational task. current efforts @xcite @xcite on simmc target this task by designing unique model structures, auxiliary losses, and pre-training strategies. although their approaches could improve the model's performance in predicting metadata related answer, such improvements partially come from shortcuts that identity object ids of "two purple dresses" 42 44 [231, 388, 319, 81] [425, 395, 367, 108] ② ③ ④ ① query brand and price based on object ids include "sleeveless and in extra large" into the search criteria. search for objects in the whole scene ① ② ③ ④ metadata multi-step reasoning … dialogue scene [after reasoning] system: the purple dress on the right is what i have to show you like that. [before reasoning] system: which items are you interested in learning more about? figure 1 : the multi-modal multi-step reasoning process for complex user requirements in the real world. numbers ❶❷❸❹ represent the sequence of reasoning steps. to make the response to user, the situated conversational agent needs to identify objects on the scene image, query information from metadata, add extra user requirements and search for the target object in the whole scene. without such a reasoning process, the agent fails to give an effective answer. to this end, inspired by cognitive theory, we propose a step-by-step multi-modal reasoning framework. concretely, in our framework, system 1's implicit understanding is handled by the conventional encoder-decoder, while the explicit step-bystep reasoning of system 2 is handled by the newborn reasoner. besides, we propose a simple, unsupervised step-by-step reasoning process generation approach based on rewriting the original user utterance as a learning signal for the reasoner. we evaluate our model on the 11th dialog system technology challenge (dstc-11) simmc 2.1 @xcite track. simmc 2.1 includes four sub-tasks of situated conversational task, i.e., ambiguous candidate identification (task 1 ambigu.), multi-modal coreference resolution (task 2 mm-coref), multi-modal dialog state track-ing (task 3 mm-dst) and multi-modal dialog response generation (task 4 res. gen.). notably, after the official evaluation of the dstc-11 committee, our proposed model achieved the 1st rank on the summing scores across all four sub-tasks. ## method our model contains a dual system, similar to the human cognitive system, consisting of system 1 and system 2. system 1 consists of a conventional encoder-decoder, which is dedicated to a specific task. system 2 consists of a generative reasoner capable of producing step-by-step multi-modal reasoning path generation. ## conclusion in this paper, we propose to enhance the ability of situated conversational agents by emphasizing step-by-step multi-modal reasoning. specifically, the original user utterance is rewritten into a step-by-step reasoning process by introducing a knowledge-and semantic-based extraction approach. in light of dual process theory, we propose a step-by-step multi-modal reasoning framework, and transmit complicated multi-step reasoning and implicit multi-modal comprehension to different modules. extensive experiments on benchmarks simmc validate the superiority of our framework. after the official evaluation of the 11th dialog system technology challenge (dstc-11) simmc 2.1 track, our model is ranked at the 1st of the summing score of all sub-tasks.