The S tat C an Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents (2023)

Abstract: We introduce the StatCan Dialogue Dataset consisting of 19,379 conversation turns between agents working at Statistics Canada and online users looking for published data tables. The conversations stem from genuine intents, are held in English or French, and lead to agents retrieving one of over 5000 complex data tables. Based on this dataset, we propose two tasks: (1) automatic retrieval of relevant tables based on a on-going conversation, and (2) automatic generation of appropriate agent responses at each turn. We investigate the difficulty of each task by establishing strong baselines. Our experiments on a temporal data split reveal that all models struggle to generalize to future conversations, as we observe a significant drop in performance across both tasks when we move from the validation to the test set. In addition, we find that response generation models struggle to decide when to return a table. Considering that the tasks pose significant challenges to existing models, we encourage the community to develop models for our task, which can be directly used to help knowledge workers find relevant tables for live chat users.

## introduction one of the longstanding goals in natural language processing (nlp) is to develop conversational agents that assist people with concrete tasks, such as finding information in large collections of documents or booking restaurants and hotels. to aid the development of such virtual assistants, the research community is in need of benchmarks that reflect the intents and linguistic phenomena found in real-world applications. however, developing such real-world conversational datasets is challenging in the current research landscape. on the one hand, academic labs often struggle to come up with sourcing from these live chats, we present the statcan dialogue dataset, a collection of 20k+ english and french conversations between visitors of statcan.gc.ca and agents working at statistics canada. before releasing this dataset, statcan has ran several procedures to remove personally identifiable information (pii). while we observe a wide variety of user intents, ranging from table manipulation to navigation instructions, a large number of visitors use the chat functionality to find data tables on the statcan website. specifically, we observe 6.6k instances where agent returns a link to a data table across 4.4k conversations. in in this work, we develop two novel tasks centered on helping users find specific tables. first, we introduce the table retrieval task, which requires a model to predict the table returned by the agent given the messages sent so far. second, we introduce the response generation task, which requires a model to predict the agent's response given the dialogue history. for both tasks, we investigate its difficulty by establishing strong baselines and evaluating them on various metrics. we stress that both tasks are immediately useful in a real-world setting. the table retrieval task can help agents find relevant tables faster while the response generation task may lead to a virtual agent that can return relevant tables through an online conversation. we hope that this tight connection with a real-world scenario will bring the research community more insight into the challenges of developing practical dialogue agents and lead to faster transfer of research ideas and findings. ## related work this section presents various directions related to our work. see open-domain qa this is the task of answering questions using a large and diverse collection of text documents. one of the first large-scale evaluations in open-domain qa was presented at trec-8 @xcite . since then, many studies have released large-scale open-domain qa datasets: wikiqa @xcite and ms marco @xcite source questions from the bing search engine, natural questions (nq) @xcite from google search, and dureader @xcite source questions in chinese from baidu. the questions come from real users and the answers are collected from the search results through crowd workers. although those datasets have questions with genuine intent and the answer must be retrieved from a collection of documents, our dataset emphasizes the retrieval of tables (in a conversational setting) rather than free-form documents. table retrieval and qa following works on tabular pre-training @xcite , table-to-text generation @xcite and weak supervision for semantic parsing @xcite , chen et al. (2020) and @xcite respectively propose ott-qa and tapas-nq, two novel approaches that extend open-domain qa to retrieving tables instead of documents. the former collects both the questions and answers from crowd workers and the latter extends natural questions by using tables from the article where the answer was taken. in both cases, the tables being retrieved are sourced from wikipedia articles. although our data also incorporate tabular retrieval, the tables are sourced from statcan.gc.ca, they can be significantly larger (as discussed in appendix a.2.1), and they are being retrieved in an interactive and conversational setting. conversational qa several works extended question answering to the conversational setting. coqa @xcite and quac @xcite introduced datasets in which multiple rounds of questions are asked about a reference passage taken from a document (such as a wikipedia article). subsequent works extended this setup to an open-domain setting where the reference passage is not known beforehand @xcite @xcite . @xcite proposed a conversational qa task about regulatory texts. aforementioned datasets are all structured in the same way: at every turn, the first speaker will ask a question, and the other speaker will give an answer. in contrast, the queries in our conversations are not restricted to questions, and the answers can be either a table, metadata, or a dialogue act. task-oriented dialogue our work is related to work on task-oriented dialogue where users converse with virtual agents to accomplish specific goals, such as booking a restaurant or resolving a customer issue. while early work has collected a dataset in a genuine information seeking setup @xcite , many recent datasets has collected them through a simulated setup @xcite @xcite @xcite . taskoriented models usually track the dialogue state by predicting dialogue acts that are specified through intents and slot-value pairs, e.g., findrestaurants(@xmath0. while our dataset does not provide turn-based annotations, the released conversations come with an annotated goal i.e., which data table the user was looking for. like other goal-oriented dialogue tasks, this annotation enables us to automatically evaluate the dialogue models through a task completion metric. ## dataset the statcan dialogue dataset consist of conversations collected from the live chat between march 1, 2019 till march 8, 2021. although a variety of user intents can be found in the broader dataset of over 25k conversations, we focus on a single intent by selecting all conversations where the agent returns a data table. we use this subset to develop and test models for the two tasks that we introduce in section 4. in section 3.1, we provide basic statistics about this subset of the data and present a dialogue analysis for a small number of conversations in section 3.2. in section 3.3, we turn our attention to the data tables and explain what kind of information is available for them. finally, we explain how the dataset is split into a train, validation, and test set in section 3.4. for technical specifications, a dataset card is provided in appendix b. ## tasks based on the conversational and tabular data, we propose two tasks: (i) a table retrieval task, which requires a model to use a partial conversation to predict the table an agent will return, and (ii) a response generation task, which requires a model to use a partial conversation to generate the most probable response by the agent. the conversations in the tasks are available in both english and french. ## models to help understand the performance of finetuned models on our tasks, this section presents an overview of the methods for the english splits, whereas the implementation details are covered in appendix a.4. similar architectures were used for french (described in appendix a.5). ## results and discussions based on our baselines and data, we report the results and analyze the challenges that our dataset and tasks pose for existing models. for the english splits, the main retrieval results are reported in ## conclusion in this paper, we introduce the statcan dialogue dataset, a novel corpus consisting of 20k+ english and french conversations between online visitors of statcan.gc.ca and operators of statistics canada. based on this dataset, we propose two tasks centered on helping users find specific data tables: the table retrieval task and the response generation task. for the table retrieval task, we experiment with various dpr and tapas variants, finding that dpr strongly outperforms its tapas counterpart, as well as the bm25 baseline. for the response generation task, we investigate fine-tuned t5-large models and explore variants where the input is augmented with table titles from dpr. we find that retrieval-augmented t5 models more frequently return the correct tables, although its title accuracy is still lower than the corresponding recall of the dpr retriever. this result suggests that the generation models struggle to decide when to return a table. we also find that retrieval and generation have difficulty generalizing to future conversations, as our temporal test split revealed a big performance gap between the validation and test set. all in all, we believe that our tasks pose significant challenges to currently available models and encourage the research community to further explore this dataset and build conversational models that help users of statistics canada.