P roto QA : A Question Answering Dataset for Prototypical Common-Sense Reasoning (2020)

Abstract: Given questions regarding some prototypical situation — such as Name something that people usually do before they leave the house for work? — a human can easily answer them via acquired experiences. There can be multiple right answers for such questions, with some more common for a situation than others. This paper introduces a new question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a long-running international trivia game show – Family Feud. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. We also propose a generative evaluation task where a model has to output a ranked list of answers, ideally covering all prototypical answers for a question. After presenting multiple competitive baseline models, we find that human performance still exceeds model scores on all evaluation metrics with a meaningful gap, supporting the challenging nature of the task.

## introduction humans possess the ability to implicitly reason using a wealth of common background knowledge, much of which is acquired through shared experiences. for example, consider the question in figure 1 -"name something that people usually do before they leave the house for work.". humans can agree about the details and characteristics of a prototypical event or situation @xcite due to commonalities in their shared lived experiences, cultural norms and expectations. this rough agreement extends beyond an agreement on a single top response, but can be viewed categories: printer/copier (37), office furniture (15), computer equipment (17), stapler (11) files (10), office appliances (5), security systems (1) (iii) name something that an athlete would not keep in her refrigerator. categories: unhealthy / fast food (36), unhealthy drinks (24), clothing/shoes (24) accessories (7) categories: phone (24), toothbrush/towels (17), clothing/shoes (15) keys ( 14 ), purse/wallet (14), accessories (8), charger (5) figure 1 : we focus on common-sense reasoning over prototypical situations when there could be many different answers but some are more common than others. our task is in generative style (not multiple-choice format). answers to a question are crowd-sourced from 100 workers and are then manually clustered into categories. to perform well, a model has to output a ranked list of answers covering multiple categories. as a ranked list of plausible answers, as demonstrated in figure 1 . such sets of diverse answers represent the nature of common sense knowledge and may be useful in applications such as dialogue systems, where multiple responses are appropriate for a given context @xcite . we present a new question/answer dataset capturing both the plausibility of the answers and the ranking preference of each answer about such prototypical situations inspired by the long-running american game show family-feud, which also provides the training data for the task. 1 the game show is played by prompting participants with queries such as name something that people usually do before they leave the house for work (as shown in figure 1 ). the answers to such questions are provided by 100 randomly selected individuals and clustered into general categories by a professional polling company. contestants attempt to provide an answer which matches these categories and get points according to the proportion of surveyed responses within a matched category. for example, when we polled 100 people with the same question (figure 1 ), they provided 43 answers involving showering/cleaning, 30 answers mentioning breakfast, and the remainder fell into smaller groups such as locking a door/grabbing keys, saying goodbye, and praying. in a family-feud game, if two participants on a team answered "grab a shower" and "eggs and coffee", they would receive 73 points for providing answers which matched these two large categories. we suggest that this is an appealing paradigm for such question answering tasks where a wide range of acceptable answers exist, as it encourages both highly popular answers as well as wide coverage over the range of good answers. we frame this task as a generative evaluation task in which a model outputs a ranked list of answers to a given question. each answer string is then matched to one or more clusters of reference answers for that question. matching an answer cluster gives the model a score equal to the cluster size. our evaluation metrics ( § 3) reward models which provide the most common answers, while also measuring the model's ability to provide a diverse set of answers in order to match all the answer clusters. while such an approach can penalize a correct model prediction when it does not match an existing reference answer, we counter this issue by (a) gathering and clustering a large number of reference answers, and (b) utilizing methods of matching non-exact matches, such as wordnet @xcite and contextual language models such as roberta @xcite . generative evaluation approaches are also used in other nlp tasks such as summarization @xcite and translation @xcite . we evaluate on a set of competitive baseline models -from qa models powered by large masked lms such as bert, to the direct prediction of answers in a language-modeling paradigm using a large gpt-2 lm @xcite , as well as gpt-2 fine-tuned upon the training data. while most models perform quite poorly at this challenging task, when gpt-2 was fine-tuned using the family-feud training set its performance did improved drastically, although remaining significantly below the score of human-level performance. the contributions of this paper are as follows. ## evaluation we present a number of methods for evaluating system-generated answers against these sets of clus-tered answers. in each, models are evaluated by providing a ranked list of answers in response to a question. these answers are then compared to the set of reference answers for that question and scored based upon how similar they are to the known answers. while one might instead convert questionanswer pairs into a multiple-choice paradigm by generating negatives, it is difficult to generate good negative examples, and the quality of a dataset can be compromised if such examples are either too easy or easily identified using biases in the negative example generation process @xcite @xcite @xcite . we outline here our proposed method for scoring these ranked lists of predicted answers. the dataset ground truth is a ranked list of clusters of answers, including weights(cluster sizes) associated with each cluster. a first component in such an evaluation is to match each answer to an existing cluster of answers, if any cluster is acceptable. we try both simple methods such as exact match as well as more flexible ways of matching to clusters, such as using synonyms from wordnet @xcite or a vector-based similarity method using roberta @xcite . the second component in this generative evaluation is to provide an overall score for the entire ranked list of answers by mapping individual answers to answer clusters or marking them wrong. scoring answers against clusters alone does not take into account the ranking. to that end, we propose two different metrics, one similar to hits@k in traditional information retrieval task and one which limits the number of incorrect answers, which is closer to how humans are typically evaluated on this task. in each case the score reported is calculated as a percentage of the oracle score. both proposed methods of scoring reward models which provide a name something that people usually do before they leave for work. shower (43): take a shower, shower ... breakfast (30): eat breakfast, breakfast ... dress (7): get dressed ... ## baselines we explore three baseline models for this task: a qa-based model which retrieves related posts in a discussion forum for each question, a languagemodeling baseline which examines how well modern pre-trained language models do at directly producing the answers, and a fine-tuned version of the language-model baseline. ## discussion and analysis ## related work a wide variety of common sense reasoning datasets address related topics. many datasets cover physical and spatial reasoning @xcite , social common sense @xcite , and common sense understanding of plausible sequences of events @xcite @xcite @xcite or understanding of the entailments of a sentence @xcite @xcite . there is also a long history of work in modeling scripts and frames @xcite @xcite @xcite , which is related to the current focus on prototypical situations. recent works have also sought to characterize the ability of pre-trained language models to understand common sense reasoning, showing such models perform well at common sense reasoning tasks even without fine-tuning, allowing one to explore the common sense reasoning inherent in those models @xcite . of particular relevance to the current work, @xcite explored the ability of pre-trained models to predict stereotypic tacit assumptions, generalizing about entire classes of entities with statements such as "everyone knows that a bear has ". interestingly, protoqa is not the first time family-feud has been referenced in the commonsense literature. common consensus @xcite was a web-based game created with the intention of being a self-sustaining platform to collect and validate commonsense knowledge based on human goals. prior work had established the idea of using online games to simultaneously entertain and collect commonsense ## conclusion we have presented a new common sense dataset with many novel features. the collection of a large set of raw answer strings and further clustering of these strings facilitates a generative evaluation method, enabling actual use of trained models to answer real common sense questions. the inclusion of counts over clusters of answers provides a very rich dataset for training and evaluation. as shown in in addition to the elements of this task which are appealing from a common sense modeling perspective, the inherent appeal of this task to humans opens a number of possibilities for future data collection and evaluation. millions of people have played phone-based games based upon this same premise foot_7 , and prior works have obtained valuable annotations from trivia game participants @xcite . this dataset lays the foundation for larger-scale data collection which leverages people's natural interest to encourage high-quality answers to more common sense questions.