A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models (2024)

Abstract: The self-rationalising capabilities of LLMs are appealing because the generated explanations can give insights into the plausibility of the predictions. However, how faithful the explanations are to the predictions is questionable, raising the need to explore the patterns behind them further.To this end, we propose a hypothesis-driven statistical framework. We use a Bayesian network to implement a hypothesis about how a task (in our example, natural language inference) is solved, and its internal states are translated into natural language with templates. Those explanations are then compared to LLM-generated free-text explanations using automatic and human evaluations. This allows us to judge how similar the LLM’s and the Bayesian network’s decision processes are. We demonstrate the usage of our framework with an example hypothesis and two realisations in Bayesian networks. The resulting models do not exhibit a strong similarity to GPT-3.5. We discuss the implications of this as well as the framework’s potential to approximate LLM decisions better in future work.

## introduction with the increasing capabilities of large language models (llms), more and more tasks that were traditionally solved using human experts and statistical models are now aided by llms. understanding how a model produces its output is an essential factor in the human acceptance of machine learning systems @xcite . however, understanding the connection between input and output in llms is not easily possible @xcite . recent advances in llms generating longer coherent text have popularised self-rationalising models, which produce a natural language explanation (nle) alongside their output @xcite . nles have numerous benefits over other, non-textual explanations: nles are valued more highly by human users @xcite , they can be applied to a broad range of problems and they can combine external knowledge with the model input. however, even though the nles can give insights into how plausible the predictions made by llms are, the faithfulness of the explanations to the prediction process remains at best uncertain @xcite @xcite . in this work, we propose exploring the patterns behind generated nles using a hypothesis-driven framework, with the ultimate goal of deriving a surrogate model. our framework is centred around a hypothetical global explanation (hge): a hypothesis about how the llm solves a specific task on a global, structural level. while we start off with an obviously oversimplified hypothesis to introduce and test the framework, we envision that it can be incrementally adapted to more refined hypotheses in the future. the patterns captured by each refinement step can then serve to measure their coverage, or e-recall @xcite , in the llm. the core component of our framework is a statistical surrogate model (ssm) that reflects the hge. we propose using a bayesian network @xcite with a manually designed structure as a framework for the ssm, as the bayesian network allows us to visualise the independencies among the random variables used in the ssm via a directed acyclic graph @xcite . this allows us to define the variables in the ssm and the relationships among them such that they reflect the hge. furthermore, since the structure of the ssm is based on the hge, each variable of the bayesian network is assigned a specific, semantically interpretable meaning. this allows us to generate local nles for individual inputs based on the internal state of the bayesian network. in the last step, we compare both the predicted labels and the nles produced by the ssm to those produced by the llm in order to gain insights about the faithfulness of the hge. we demonstrate the usage of this framework with an exemplary hge for how the behaviour of the llm gpt-3.5 @xcite ) can be explained when performing english-language natural language inference (nli). we discuss the challenges when designing, implementing and training the bayesian networks based on the hge and outline the next steps on the way to a surrogate that models llm predictions more accurately. self-rationalising models have received increasing attention as the generation abilities of nlp models have improved in recent years. human-annotated datasets such as e-snli @xcite for nli, @xcite and ecqa @xcite for commonsense question answering and comve @xcite for commonsense validation are the basis for much nle work @xcite . however, the role of llms in the annotation process itself is likely increasing @xcite , as it reduces the significant cost of human annotation @xcite . a concern, however, is that the generation of the nles is as opaque as the prediction process. to address this, some recent work explores the faithfulness of nles: @xcite show that prediction and explanation exhibit correlated responses to input noise. @xcite propose analysing nles after counterfactual interventions on the input that alter the prediction, and testing the sufficiency of the reasons provided in the nle. @xcite show that biased features in the input lead to obviously unfaithful nles as those features affect the predictions heavily but are never mentioned in the generated nles. similar concerns have been raised for modelagnostic surrogates such as lime @xcite and shap (lundberg and lee, 2017), which are widely used for highlighting input features as a form of explanation. they approximate a complex model by training a simpler, interpretable model on the original model's predictions. whether such surrogates are helpful for understanding complex models is subject to discussion. @xcite makes the case against their usage in high-stake decisions and argues for the deployment of models with an interpretable decision process. however, from a more practical perspective, jacovi and goldberg (2020) point out that faithfulness is not a binary feature but should be seen as a scale. they argue that a man inspects the uniform of a figure in asia. the man is sleeping. ## proposed framework in this section, we describe our framework foot_0 : how we design the ssm based on our hypothesis ( §3.1- §3.3), how we determine and learn the parameters ( §3.4) and how we generate nles based on the ssm's parameters ( §3.5). we demonstrate that we can successfully control for the assumptions made in our hypothesis, with a model that is intuitive to construct and understand. for a simplified demonstration of how the framework processes an example input, we refer to figure 1. ## comparison to the llm now that we have constructed the ssm, we need to compare it to gpt-3.5. to that end, we first generate labels and explanations with gpt-3.5 ( §4.1), and compare them to the ssm in human ( §4.2.1) and automatic ( §4.2.2) evaluations. ## results in this section, we report the experimental results using the evaluation procedures and measures introduced in section 4.2, assessing how similar our surrogate models are to gpt-3.5. for illustratory purposes, we will provide example explanations generated by the ssms in §5.1, followed by the results of the human ( §5.2) and the automatic evaluations ( §5.3). ## discussion this paper set out with the aim of constructing and evaluating a hypothesis-driven surrogate for how gpt-3.5 performs nli. as we have seen in section 3, the bayesian network allows us to incorporate our hypothesis in an intuitive and testable way, making (in-)dependencies between phrases explicit. while the framework is convenient and functional, the results in section 5 indicate a low similarity to gpt-3.5. this can have two reasons: first, that the hge does not resemble the way gpt-3.5 solves the nli task, or second, that implementation details of the ssm have shortcomings that limit the performance of the model. in any case, several challenges need to be addressed before an ssm can potentially pass as a surrogate model. we discuss the shortcomings of our models as well as paths forward in §6.1 and 6.2, and promising paths for other future research based on our insights from this paper in 6.3. ## conclusion this paper suggests a framework for testing a hypothesis about the decision-making process of llms using bayesian networks. we demonstrate how to construct a bayesian network based on a hypothetical global explanation and how to evaluate the alignment of the llm with this network. the framework intuitively implements the hypothesis as the bayesian network that it is based on can model our assumptions natively, and its random variables can be translated into natural language. our surrogates were however not similar to gpt-3.5, which can have two reasons: that the hypothesis itself does not accurately describe gpt-3.5's behaviour and needs to be replaced or complemented by other hypotheses, or that simplifications we made in our implementation affected the performance and thereby also the similarity to gpt-3.5. we assume that we are dealing with a combination of the two reasons, and suggest that working on the identified issues can lead to more accurate surrogate models that can help us understand the behaviour of llms better.