JECC : Commonsense Reasoning Tasks Derived from Interactive Fictions (2023)

Abstract: Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems. We proposea new commonsense reasoning dataset based on human’s Interactive Fiction (IF) gameplaywalkthroughs as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset provides a natural mixture of various reasoning types and requires multi-hopreasoning. Moreover, the IF game-based construction procedure requires much less humaninterventions than previous ones. Different from existing benchmarks, our dataset focuseson the assessment of functional commonsense knowledge rules rather than factual knowledge. Hence, in order to achieve higher performance on our tasks, models need to effectively uti-lize such functional knowledge to infer the outcomes of actions, rather than relying solely onmemorizing facts. Experiments show that the introduced dataset is challenging to previousmachine reading models as well as the new large language models with a significant 20%performance gap compared to human experts.

## introduction there has been a flurry of datasets and benchmarks proposed to address natural language-based commonsense reasoning @xcite @xcite @xcite @xcite @xcite @xcite . these benchmarks usually adopt a multi-choice formwith the input query and an optional short paragraph of the background description, each candidate forms a statement; the task is to predict the statement that is consistent with some commonsense knowledge facts. these benchmarks share some limitations, as they are mostly constructed to focus on a single reasoning type and require similar validation-based reasoning. first, most benchmarks concentrate on a specific facet and ask human annotators to write candidate statements related to the particular type of commonsense. as a result, the distribution of these datasets is unnatural and biased to a specific facet. for example, most benchmarks focus on collocation, association, or other relations (e.g., conceptnet @xcite relations) between words or concepts @xcite @xcite . other examples include temporal commonsense @xcite , physical interactions between actions and objects @xcite , emotions and behaviors of people under the given situation @xcite , and cause-effects between events and states @xcite @xcite . second, most datasets require validation-based reasoning between a commonsense fact and a text statement but neglect hops over multiple facts. 2 the previous work's limitations bias the model evaluation. for example, pre-trained language models (plms), such as bert @xcite , can well handle most benchmarks, because their pre-training process may include texts on the required facts thus provide shortcuts to a dominating portion of commonsense validation instances. in summary, the above limitations of previous benchmarks lead to discrepancies among practical nlp tasks that require broad reasoning ability on various facets. our contribution. we derive a new commonsense reasoning dataset from the model-based reinforcement learning challenge of interactive fictions (if) to address the above limitations. recent advances @xcite @xcite in if games have recognized several commonsense reasoning challenges, such as detecting valid actions and predicting different actions' effects. figure 1 illustrates sample gameplay of the classic game zork1 and the required commonsense knowledge. we derive a commonsense dataset from human players' gameplay records related to the second challenge, i.e., predicting which textual observation is most likely after applying an action or a sequence of actions to a given game state. the derived dataset naturally addresses the aforementioned limitations in previous datasets. first, predicting the next observation naturally requires various commonsense knowledge and reasoning types. as shown in figure 1 , a primary commonsense type is spatial reasoning, e.g., "climb the tree" makes the protagonist up on a tree. another primary type is reasoning about object interactions. for example, keys can open locks (object relationships); "hatch egg" will reveal "things" inside the egg (object properties); "burn repellent" leads to an explosion and kills the player (physical reasoning). the above interactions are more com-2 some datasets include a portion of instances that require explicit reasoning capacity, such as @xcite @xcite . but still, standalone facts can solve most such instances. finally, compared to previous works that heavily rely on human annotation, our dataset construction requires minimal human effort, providing great expansibility. for example, with large amounts of available if games in dungeon crawls, sci-fi, mystery, comedy, and horror, it is straightforward to extend our dataset to include more data samples and cover a wide range of genres. we can also naturally increase the reasoning difficulty by increasing the prediction horizon of future observations after taking multi-step actions instead of a single one. in summary, we introduce a new commonsense reasoning dataset construction paradigm, collectively with two datasets. the larger dataset covers 29 games in multiple domains from the jericho environment @xcite , named the jericho environment commonsense comprehension task (jecc). the smaller dataset, aimed for the single-domain test and fast model development, includes four if games in the zork universe, named zork universe commonsense comprehension (zucc). we provide strong baselines to the datasets and categorize their performance gap compared to human experts. ## related work previous work has identified various types of commonsense knowledge humans master for text understanding. as discussed in the introduction section, most existing datasets cover one or a few limited types. also, they mostly have the form of commonsense fact validation based on a text statement. semantic relations between concepts. most previous datasets cover the semantic relations between words or concepts. these relations include the concept hierarchies, such as those covered by wordnet or conceptnet, and word collocations and associations. for example, the early work winograd @xcite evaluates the model's ability to capture word collocations, associations between objects, and their attributes as a pronoun resolution task. the work by @xcite is one of the first datasets covering the conceptnet relational tuple validation as a question-answering task. the problem asks the relation of a source object, and the model selects the target object that satisfies the relation from four candidates. @xcite focus on the collocations between adjectives and objects. their task takes the form of textual inference, where a premise describes an object and the corresponding hypothesis consists of the object that is modified by an adjective. @xcite ) study associations among multiple words, i.e., whether a word can be associated with two or more given others (but the work does not formally define the types of associations). they propose a new task format in games where the player produces as many words as possible by combining existing words. ## dataset construction: commonsense challenges from if games we pick games supported by the jericho environment @xcite to construct the jecc dataset. 4 we pick games in the zork universe for the zucc dataset. 5 we first introduce the necessary definitions in the if game domain and then describe how we construct our zucc and jecc datasets as the forward prediction tasks based on human players' gameplay records, followed by a summary on the improved properties of our dataset compared to previous ones. the dataset will be released for public usage. it can be created with our released code with mit license. ## neural inference baselines we formulate our task as a textual entailment task that the models infer the next state o t+1 given o t and a t . we provide strong textual entailmentbased baselines for our benchmark. we categorize the baselines into two types, namely pairwise textual inference methods and the triplewise inference methods. the notations o t , a t of observations and actions represent their word sequences. ## experiments we first evaluate all the proposed baselines on our datasets. then we conduct a human study on a subset of our development data to investigate how human experts perform and the performance gap between machines and humans. ## conclusion interactive fiction (if) games encode plentiful and diverse commonsense knowledge of the physical world. in this work, we derive commonsense reasoning benchmarks jecc and zucc from if games in the jericho environment. taking the form of predicting the most likely observation when applying an action to a game state, our automatically generated benchmark covers comprehensive commonsense reasoning types such as spatial reasoning and object interaction, etc. our experiments show that current popular neural models have limited performance compared to humans. to our best knowledge, we do not identify significant negative impacts on society resulting from this work.