QUITE : Quantifying Uncertainty in Natural Language Text in B ayesian Reasoning Scenarios (2024)

Abstract: Reasoning is key to many decision making processes. It requires consolidating a set of rule-like premises that are often associated with degrees of uncertainty and observations to draw conclusions. In this work, we address both the case where premises are specified as numeric probabilistic rules and situations in which humans state their estimates using words expressing degrees of certainty. Existing probabilistic reasoning datasets simplify the task, e.g., by requiring the model to only rank textual alternatives, by including only binary random variables, or by making use of a limited set of templates that result in less varied text.In this work, we present QUITE, a question answering dataset of real-world Bayesian reasoning scenarios with categorical random variables and complex relationships. QUITE provides high-quality natural language verbalizations of premises together with evidence statements and expects the answer to a question in the form of an estimated probability. We conduct an extensive set of experiments, finding that logic-based models outperform out-of-the-box large language models on all reasoning types (causal, evidential, and explaining-away). Our results provide evidence that neuro-symbolic models are a promising direction for improving complex reasoning. We release QUITE and code for training and experiments on Github.

## introduction reasoning about causality is an integral part of intelligence, as it helps to understand and predict the world. in the real world, causes and associations can rarely be determined with complete certainty, and reasoning becomes inherently difficult if uncertainties are involved @xcite ). an automated system for interpreting text describing causal relationships and their associated numeric probabilities or verbalized degrees of uncertainty would be highly useful in domains such as requirements engineering @xcite or text-mining in clinical documentation @xcite . modeling linguistically expressed uncertainty has been an active research area for decades in natural language processing (nlp) @xcite @xcite . recently, large language models (llms) have shown superior performance on many nlp tasks. however, they fall short of incorporating principled reasoning mechanisms, with frequent failure cases @xcite , and their mathematical skills decline if presented with unseen cases @xcite . in zero-shot or chain-of-thought (cot) prompting settings, open-source and open-weights llms are also unable to outperform a random baseline in bayesian inference in higher-order causal networks @xcite . gpt-3 and gpt-4 are somewhat better, yet do not excel at the task. to probe llms for their reasoning capabilities, @xcite compile the cladder dataset based on toy causal inference scenarios taken from textbooks and literature on causal reasoning. cladder evaluates performance by rungs of the ## background and related work in this section, we start by introducing the basic concepts of bayesian networks and corresponding reasoning types, closely following the terms and definitions of @xcite . we then review existing literature on modeling uncertainty in language, benchmark datasets for bayesian reasoning, and semantic parsing to logical forms. ## dataset in this section, we describe the dataset creation process of quite, provide dataset statistics, assess its linguistic quality, and validate the translations of the logical structures into natural language. ## modeling in this section, we describe the various models that we benchmark using quite. fine-tuned models are suffixed -ft in the following. ## experimental evaluation in this section, we report our extensive evaluation of current state-of-the-art llms and our neurosymbolic model on quite. all fine-tuning details and parameters are provided in appendix c. ## conclusion and outlook in this paper, we have presented quite, a new question answering dataset that provides bayesian reasoning scenarios for a variety of domains and that can be used to assess uncertainty-based reasoning with llms. from a large set of experiments using numeric probabilistic premises and premises expressed using words of estimative probability, we conclude that a neuro-symbolic approach combining probabilistic logic programming and finetuned llms as semantic parsers is most promising. moreover, we find that non-specialized llms mostly fail on this task. outlook. for increasing the robustness of logicbased semantic parsing models, different approaches should be further investigated. for example, constrained decoding techniques could be used to ensure that only valid predicates can be generated at any point in time. next steps include also studying reasoning in modal and counterfactual scenarios.