  We present a new corpus for the Situated and Interactive Multimodal
Conversations, SIMMC 2.0, aimed at building a successful multimodal assistant
agent. Specifically, the dataset features 11K task-oriented dialogs (117K
utterances) between a user and a virtual assistant on the shopping domain
(fashion and furniture), grounded in situated and photo-realistic VR scenes.
The dialogs are collected using a two-phase pipeline, which first generates
simulated dialog flows via a novel multimodal dialog simulator we propose,
followed by manual paraphrasing of the generated utterances. In this paper, we
provide an in-depth analysis of the collected dataset, and describe in detail
the four main benchmark tasks we propose for SIMMC 2.0. The preliminary
analysis with a baseline model highlights the new challenges that the SIMMC 2.0
dataset brings, suggesting new directions for future research. Our dataset and
code will be made publicly available.
