Navigating Connected Memories with a Task-oriented Dialog System (2022)

Abstract: Recent years have seen an increasing trend in the volume of personal media captured by users, thanks to the advent of smartphones and smart glasses, resulting in large media collections. Despite conversation being an intuitive human-computer interface, current efforts focus mostly on single-shot natural language based media retrieval to aid users query their media and re-live their memories. This severely limits the search functionality as users can neither ask follow-up queries nor obtain information without first formulating a single-turn query.In this work, we propose dialogs for connected memories as a powerful tool to empower users to search their media collection through a multi-turn, interactive conversation. Towards this, we collect a new task-oriented dialog dataset COMET, which contains 11.5k user↔assistant dialogs (totalling 103k utterances), grounded in simulated personal memory graphs. We employ a resource-efficient, two-phase data collection pipeline that uses: (1) a novel multimodal dialog simulator that generates synthetic dialog flows grounded in memory graphs, and, (2) manual paraphrasing to obtain natural language utterances. We analyze COMET, formulate four main tasks to benchmark meaningful progress, and adopt state-of-the-art language models as strong baselines, in order to highlight the multimodal challenges captured by our dataset.

## introduction the rise of smartphones and smart glasses has contributed to a surge in the amount of personal media (photos, videos, montages, etc.) captured by users on a day-to-day basis in the past decade. for instance, it is estimated that about 1.5 trillion photos would be clicked in the year 2022 @xcite . as a result, personal media collections typically grow at an alarming rate, making it cumbersome for users to manually search, retrieve, and re-live their captured memories foot_0 . to alleviate this situation, solutions that perform natural language query-based media retrieval @xcite @xcite @xcite @xcite have been proposed. however, such approaches exhibit two drawbacks. first, they are single-shot interactions without any context carry-over, e.g., show me some photos from the beach last week.. this limits the functionality and does not let users ask any follow-up queries like 'display photos from the first time i was here?', since understanding here requires the query history. second, users cannot seek information without actually formulating the query to retrieve the corresponding memory. for instance, there is no easy query to know the first time a user visited the beach in the memory they are reviewing. in order to overcome these limitations, we propose dialogs for connected memories as a powerful interface where users can interactively query their memory collections. by design, a conversational agent can handle multi-turn interactions enabling several additional queries that require context carryover, e.g., 'when was the first time i was at this beach?'. though prior efforts have explored the use of dialogs in media retrieval @xcite in other domains (e.g., fashion), there is no existing work focusing on interactive search and query of personal media collections to the best of our knowledge. more concretely, we propose comet, a new multimodal task-oriented dialog dataset aimed at developing conversational assistants that can enable users to interactively search and query their collection of memories. working with personal media collections presents two main obstacles: (a) there are no readily available public datasets that contain personal media along with associated media attributes that we could leverage, and, (b) personal memories constitute sensitive information, thus resulting in privacy and safety concerns. to circumvent these roadblocks, we devise a novel memory graph simulator that can leverage publicly available media datasets and help create several synthetic memory collections. we represent these collections as memory graphs to capture useful relationships between the constituent memories, e.g., memories taken at the same place. we then collect 11.5k user↔assistant task-oriented dialogs (totalling 103k utterances), grounded in 1.1k memory graphs. an example dialog is in fig. 1 . our dataset is challenging as it requires reasoning through both the dialog history and multimodal context (memory graphs) to resolve coreferences, track the dialog state, predict the right api, and generate a meaningful natural language assistant response. as an example, consider the query 'when was the first time i was here?'. first, the model needs to resolve here using the dialog history and previously viewed memories. next, it needs to understand that the query is seeking in-formation about a connected memory, and predict the right api get_time(resolve(here), first time). finally, it should produce a response like 'the first time you were here was on august 2, 2019 with jean', potentially including some chit-chat. to capture these challenges and benchmark progress towards assistants that can interactively handle dialogs for connected memories, we formulate four main tasks: assistant api call prediction, multimodal coreference resolution (mm-coref), multimodel dialog state tracking (mm-dst), and response generation. we train baseline models for these tasks, and discuss future research directions. the most notable modeling approaches for taskoriented dialog systems include casting the dst task as a joint causal language modeling problem @xcite @xcite , by fine-tuning a large pre-trained transformers. we follow this recent trend and provide baselines by extending it accommodate for the unique multimodal contexts that our dataset brings. multimodal dialogs have become increasingly more popular, where dialog models process both visual and text input to handle queries @xcite . many existing literature @xcite @xcite @xcite study multimodal q&a dialogs grounded on a single image as grounding context, extending the conventional vqa @xcite tasks to multi-turn scenarios. @xcite studies conversational recommendation system using personal memories. we extend this line of work by studying the multimodal agent that operates on a collection of media (memory graph), thus requiring reasoning abilities over multiple grounding contexts. our focus on taskoriented dialogs extends the previous literature and datasets that primarily focus on retrieval tasks @xcite @xcite ), capturing structured user intents and fine-grained attributes for each multimodal query. memory qa: our work is also similar to the memory qa tasks @xcite , where the main task is to answer user qa queries upon a collection of images, extending the visual qa task @xcite which operates on a single image. however, the existing literature is limited to a simple single-turn qa interaction, and focuses on the identification of an evidential image to answer a question. while our dataset does include qa queries, we extend the problem domain to the conversational settings which support complex scenarios (e.g. searching for related memories), allowing for rich multimodal interactions. ## the comet dataset comet is aimed to enable assistant systems that can process interactive queries from users and help navigate their collection of memories through a natural language conversation. towards this, we collect the comet dataset using a two-phase approach (shown in fig. 2 ): (a) generating synthetic dialog flows between a user and an assistant that are conditioned on memory graphs, using a novel multimodal dialog simulator (sec. 3.1), and, (b) manually paraphrasing the above flows to obtain dialogs with natural language utterances (sec. 3.2), thus moving closer to real-world application. this approach is resource-efficient as it reduces the annotation overheads when compared to collecting human↔human dialogs, both in terms of cost and time. in what follows, we describe these two phases in detail and analyze our comet dataset. see the supplementary (fig. 7 ) for an example dialog. ## modeling & empirical analysis we now perform preliminary empirical evaluation and analysis for the proposed tasks by training baselines. detailed modeling work is left as future work. dataset split. the dataset is randomly divided into: train (70%), val (15%), and test (15%). for our experiments, models are trained using train split and performance is reported on test, while val is used to pick the model hyper-parameters. notations. we follow the notation established in @xcite , where each dialog of length n r rounds is represented as @xmath0, a i , m i , b i )} nr @xmath1: • u i : user utterance at turn i • a i : assistant utterance at turn i • m i : multimodal context, i.e., memory graph and memories retrieved in the previous turns, • b i : multimodal belief state, a semantic parse of u i (intent, slot, memory references). therefore, given the current user utterance (u t ), dialog history h @xmath2, a i ) t-1 @xmath3, and the multimodal context (m t ), a comet agent should predict the user belief state b t and the natural language response a t for every dialog turn t. baselines. causal language models pretrained on large datasets have shown a lot of promise in multimodal and text-only task-oriented dialog modeling, when finetuned on the downstream task @xcite @xcite . following this popular approach, we adopt the transformer-based gpt-2 @xcite model and jointly train it for api prediction, mm-coref, dst, and response generation tasks, as shown in fig. 6 . in particular, we use the 12-layer gpt-2 (117m ) model and finetune it on dialogs from comet dataset, using early stopping based on token perplexity (<3 gpu hrs). we use two approaches to capture m i : (a) text-only (gpt2-text), where previously viewed memories and their attributes are represented as flattened strings. note that this baseline uses groundtruth activities from the memory graph. (b) multimodal (gpt2-mm), where bottom-up and top-down (butd) @xcite and clip @xcite image features are extracted for previous viewed memories, and fed as 'visual tokens' while finetuning the gpt-2 model. analysis. a key observation from tab. 3 is that multimodal models outperforms text-only across all the tasks significantly. this is intuitive, for instance, multimodal coreference resolution requires understanding the memories beyond the obvious activity label in order to rightly resolve the reference. consider the query: 'when was the last time i played with my dog here?'. to resolve to the right memory, the system needs to understand which memory is about playing with the dog towards which a mere activity label throwing frisbee might be insufficient. for a similar reason, additional multimodal features improve response generation, especially to include chit-chat. on the other side, gpt-text performs competitively on capturing the dialog state. conclusion. we present a novel dataset for the dialogs for connected memories, comet, with 11.5k user↔assistant dialogs (103k utterances) grounded on the memory graphs. we present a novel multimodal dialog simulator, which generates simulated dialogs grounded on diverse memory graphs that are automatically configured. our empirical analysis demonstrates many new challenges that our comet dataset brings, highlighting new directions of research in this area. ## limitations the generalizability of comet is naturally bounded by the underlying graph simulator, especially around memory attribute labels of place, people, and time. however, we justify this as follows: (a) recall that the focus of our work is to enable an assistant that can understand and execute user queries about connected memories through an interactive dialog. even with the simulated dialog flows, comet captures several interesting challenges related to multimodal dialog, for instance, coreference resolution and dialog state tracking (as seen in sec. 3.3 and sec. 5). this opens the door to new research directions in multimodal conversation, especially in the absence of a readily available large-scale personal photo collection dataset (along with attributes and metadata). (b) due to the twostage data collection pipeline, comet is amenable to data augmentation techniques that can increase the robustness of the downstream dialog model. for instance, the dataset can be easily augmented by replacing named entities in the memory graph and utterances, without changing the flow.