P ragmati CQA : A Dataset for Pragmatic Question Answering in Conversations (2023)

Abstract: Pragmatic reasoning about another speaker’s unspoken intent and state of mind is crucial to efficient and effective human communication. It is virtually omnipresent in conversations between humans, e.g., when someone asks “do you have a minute?”, instead of interpreting it literally as a query about your schedule, you understand that the speaker might have requests that take time, and respond accordingly. In this paper, we present PragmatiCQA, the first large-scale open-domain question answering (QA) dataset featuring 6873 QA pairs that explores pragmatic reasoning in conversations over a diverse set of topics. We designed innovative crowdsourcing mechanisms for interest-based and task-driven data collection to address the common issue of incentive misalignment between crowdworkers and potential users. To compare computational models’ capability at pragmatic reasoning, we also propose several quantitative metrics to evaluate question answering systems on PragmatiCQA. We find that state-of-the-art systems still struggle to perform human-like pragmatic reasoning, and highlight their limitations for future research.

## introduction reasoning about interlocutors' unspoken intent or state of mind is a crucial feature of human communication, which allows us to convey ideas and exchange information more efficiently and effectively assuming that conversation participants are cooperative @xcite . for instance, when asked "is there water on mars?", a friendly, knowledgeable person will not answer just "yes". typically, the answerer would anticipate reasonable follow-up questions and/or identify the asker's theme of curiosity, * these authors contributed equally. pq designed the collection mechanism and evaluation metrics, and wrote the paper; nd carried out most of the groundwork for collection/experiments/analyses. cm/jh offered project guidance and support, and helped discuss experiments and analysis. ## pragmaticqa: pragmatic question answering in conversations in this section, we introduce how we crowdsource pragmaticqa ranging from data source and processing to details about task design and how it helps align crowdworker interest with that of potential users. we conclude the section with evaluation metrics we propose to assess the pragmatic reasoning of qa systems, as well as statistics of the dataset. ## discussion: evaluation on pragmaticqa while the proposed evaluation metrics are useful to provide quality estimates of the provided answers, especially when it comes to how well the prediction matches the annotators' pragmatic behavior, we acknowledge that this is far from a complete set of evaluations desirable for real-world systems developed on pragmaticqa to be useful. first, the evaluation metrics presented do not evaluate the final system response on its factuality or faithfulness to the spans selected, which is crucial for real-world systems. @xcite recently report that publicly available generative search engines are still far from satisfactory on this front, which we speculate will be more challenging for pragmatic responses such as those in prag-maticqa. second, unlike the literal answer, the definition of pragmatic responses in an information-seeking conversation is open-ended and more subjective in nature. in this paper, we explored categorizing these into two broad categories, answers to address potentially unspoken information needs, and potential relevant knowledge that can be helpful, but this is far from comprehensive, since good pragmatic responses could involve clarification questions that are not covered by pragmaticqa. even within these categories, we see that at a given point in the conversation, there are typically more than one follow-up questions to be asked given the literal response; relevant knowledge is only more diverse. while a high f prag 1 score can approximate a sufficient condition for a pragmatic natural language system, it might be far from necessary due to the potential existence of multiple good answers. both of these suggest that additional evaluation metrics are necessary for pragmaticqa, which we leave to future work. we believe, given these observations, that model-based evaluation will become crucial in the pursuit of better evaluation methods on pragmaticqa, where our dataset will provide the resource to help kickstart the exploration. in the meantime, we also believe that the metrics presented in this paper can still serve as good proxies for evaluating model's pragmatic behavior until more powerful evaluation methods are available. ## conclusion we presented pragmaticqa, the first opendomain conversational question answering dataset featuring pragmatic answers and quantitative metrics to evaluate pragmatic reasoning in conversational qa. pragmaticqa is collected with innovative crowdsourcing techniques, including techniques that better align crowd worker incentive with eventual users of convqa systems that improves crowd worker engagement and data quality. finally, we show in our experiments that questions in pragmaticqa present unique and important challenges to convqa systems today, and open new research directions for investigation.