Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rules (2021)

Abstract: One of the challenges faced by conversational agents is their inability to identify unstated presumptions of their users’ commands, a task trivial for humans due to their common sense. In this paper, we propose a zero-shot commonsense reasoning system for conversational agents in an attempt to achieve this. Our reasoner uncovers unstated presumptions from user commands satisfying a general template of if-(state), then-(action), because-(goal). Our reasoner uses a state-of-the-art transformer-based generative commonsense knowledge base (KB) as its source of background knowledge for reasoning. We propose a novel and iterative knowledge query mechanism to extract multi-hop reasoning chains from the neural KB which uses symbolic logic rules to significantly reduce the search space. Similar to any KBs gathered to date, our commonsense KB is prone to missing knowledge. Therefore, we propose to conversationally elicit the missing knowledge from human users with our novel dynamic question generation strategy, which generates and presents contextualized queries to human users. We evaluate the model with a user study with human users that achieves a 35% higher success rate compared to SOTA.

## introduction conversational agents are becoming prominent in our daily lives thanks to advances in speech recognition, natural language processing and machine learning. however, most conversational agents still lack commonsense reasoning, preventing them from engaging in rich conversations with humans. recently, @xcite proposed a commonsense reasoning benchmark task for conversational agents that contains natural language commands given to an agent by humans. these commands follow a general template of: "if (state holds), then (perform action ), because (i want to achieve goal )". we refer to commands satisfying this template as if-thenbecause commands. as stated in @xcite , humans often under-specify conditions on the if-portion ( state ) and/or then-portion ( action ) of their commands. these underspecified conditions are referred to as commonsense presumptions. for example, consider the command, "if it's going to rain in the afternoon ( • ) then remind me to bring an umbrella ( • ) because i want to remain dry", where ( • ) indicates the position of the unstated commonsense presumptions. the presumptions for this command are (and i am outside) and (before i leave the house), respectively. the goal in this task is to infer such commonsense presumptions given if-then-because commands. in this paper, we propose the conversational multi-hop reasoner (¼ ¾) for this task which performs zero-shot reasoning. ¼ ¾ extracts a multi-hop reasoning chain that indicates how the the parser, the multi-hop reasoner (prover) and the dialog system. given an if-then-because command, the parser extracts independent natural language clauses for the state , action and goal . the prover extracts multi-hop reasoning chains given the logic templates using our neural commonsense kb, comet that indicates how the action achieves the goal when the state holds. the extracted reasoning chains go through the dialog system that generates template-dependent questions and converses with a human who either validates the returned proofs or contributes novel commonsense knowledge if the proofs are incorrect. our multi-hop reasoner uses a state-of-the-art (sota) transformer-based commonsense knowledge model called commonsense transformers (comet ) @xcite as its source of background knowledge and is the first time reasoning task is tackled using a large scale kb. knowledge models can be used in place of kbs, but they are more flexible in terms of information access, so we use this term interchangeably with kb. despite being a sota kb, comet still misses requisite knowledge for reasoning about if-thenbecause commands. in fact, many if-then-because commands often fall under the long tail of tasks for which there is too little knowledge available in any commonsense kbs. for example, one of the commands in the dataset is: "if i get an email with subject the gas kiln fired, then send me a list of all the pots i put on the glaze shelf between the last firing and now, because i want to pick up the pots from the studio." it is unlikely for any knowledge source to contain a fact that is contextually relevant to this command. more importantly, it is also unlikely that a command requiring the same type of reasoning will occur again in the future, making it cost-ineffective to manually annotate. to overcome this, we propose conversationally eliciting missing knowledge from human users. conversation with users is readily available since ¼ ¾ is developed for conversational agents. we develop a novel question generation strategy that uses comet to generate questions in the context of the input if-then-because commands, allowing ¼ ¾ to acquire contextual feedback from humans. we evaluate our reasoner by conducting a user study with humans. ## background and notation in this section, we briefly re-introduce the benchmark task, the logic templates, and arabshahi et al.'s reasoning engine corgi (commonsense reasoning by instruction), along with an overview of comet @xcite . ## experiments and results here we discuss our experimental setup, evaluation method, baselines and results. we work with 132 out of the 160 commands from the benchmark dataset (2021) that fall under three logic templates blue, orange and green (figure 3 ). the red template contains commands for which there is no unifying logic template. therefore, we cannot use it. evaluation: we do not use automated evaluation metrics and instead use human evaluations for two reasons. first, there are currently no metrics in the literature that assess whether the returned reasoning chains are "correct" from a commonsense perspective. second, evaluating dialog systems is challenging. it is debated that metrics such as bleu @xcite and perplexity often fail to measure true response quality @xcite . experiments: in the first experiment, we evaluate our multi-hop prover in isolation and without conversational interactions. the human evaluators in this study are expert evaluators. in our second experiment, we test ¼ ¾'s performance end-toend with non-expert human users and investigate the efficacy of the conversational interactions. in order to be comparable with @xcite 's study, we use the knowledge base of commonsense facts and rules, k, released with the dataset. it contains 228 facts and rules. our comet model is pre-trained on knowledge graphs conceptnet and atomic. we used comet's open-source code foot_0 for training with the hyper-parameters reported there @xcite . ## related work efforts in developing commonsense reasoning started as early as the foundation of the field of artificial intelligence @xcite @xcite . commonsense reasoning is becoming more prominent as computers increase their interactions with us in our daily lives. for example, conversational agents such as alexa, siri, google home and others have very recently entered our daily lives. however, they cannot currently engage in natural sounding conversations with their human users mainly due to lack of commonsense reasoning. moreover, they operate mostly on a pre-programmed set of tasks. on the other hand, instructable agents @xcite @xcite @xcite @xcite , can be taught new tasks through natural language instructions/demonstrations. one of the challenges these bots face is correctly grounding their natural language instructions into executable commands. our approach addresses a new reasoning task proposed by @xcite that contains commands given to an instructable agent satisfying a general template. in contrast to this challenging task and timetravel @xcite , most commonsense reasoning benchmarks have traditionally been designed in a multiple choice manner. moreover, they are not typically targeted at conversational agents. refer to @xcite and @xcite for a comprehensive list of commonsense reasoning benchmarks. our commonsense reasoning engine uses a sota neural knowledge model, comet @xcite , as an underlying source of commonsense knowledge. comet is a framework for constructing knowledge bases from transformerbased language models. in contrast to previous au-tomatic knowledge base construction methods that rely on semi-structured @xcite @xcite and unstructured @xcite @xcite text extraction, comet uses transfer learning to adapt language models to generate knowledge graph tuples by learning on examples of structured knowledge from a seed kb. comet was recently used for persona-grounded dialog for chatbots @xcite . ## conclusions we introduce the conversational multi-hop reasoner (¼ ¾) for commonsense reasoning in conversational agents. ¼ ¾ uses a neural commonsense kb and symbolic logic rules to perform multi-hop reasoning. it takes if-(state), then-(action), because(goal) commands as input and returns a multi-hop chain of commonsense knowledge, indicating how the action leads to the goal when the state holds. the symbolic logic rules help significantly reduce the multi-hop reasoning search space and improve the quality of the generated commonsense reasoning chains. we evaluate ¼ ¾ with a user study with human users.