Properties and Challenges of LLM -Generated Explanations (2024)

Abstract: The self-rationalising capabilities of large language models (LLMs) have been explored in restricted settings, using task-specific data sets.However, current LLMs do not (only) rely on specifically annotated data; nonetheless, they frequently explain their outputs.The properties of the generated explanations are influenced by the pre-training corpus and by the target data used for instruction fine-tuning.As the pre-training corpus includes a large amount of human-written explanations “in the wild”, we hypothesise that LLMs adopt common properties of human explanations.By analysing the outputs for a multi-domain instruction fine-tuning data set, we find that generated explanations show selectivity and contain illustrative elements, but less frequently are subjective or misleading.We discuss reasons and consequences of the properties’ presence or absence. In particular, we outline positive and negative implications depending on the goals and user groups of the self-rationalising system.

## introduction self-rationalising models produce explanations together with their primary output, often in natural language @xcite . these models have received increased attention in recent years as language generation abilities have improved with autoregressive transformer @xcite architectures, pioneered by the gpt models @xcite . natural language explanations are easily accessible to users and flexible in the tasks they can be used for and the types of reasoning they can express. so far, the focus of this line of research has been on models trained on annotated explanations for (more or less) well-defined tasks such as commonsense question answering @xcite @xcite or natural language inference @xcite . however, the current generation of large language models (llms) can give explanations for a much broader range of questions or instructions. generated explanations can be a means to improve model performance @xcite and decrease hallucinations via a feedback loop @xcite ; but they are also expected to provide context for human @xcite . as llms typically are not explicitly trained with annotated explanations, in contrast to earlier models, the properties of the explanations they provide are not obvious, making it hard to predict the usefulness of these models' self-rationalising capabilities. two main factors can influence the explanations given by llms: the properties of the explanations contained in the pre-training data, and the properties fostered by alignment techniques such as instruction fine-tuning (ift; @xcite and reinforcement learning with human feedback @xcite . based on this, we hypothesise that llms capture various properties of human explanations from the large amount of human text in the training data, including characteristics uncommon in the earlier annotated explanations, and in particular properties that contribute primarily to the communicative function of human explanations @xcite . many of these properties have been argued to be irrelevant or even detrimental to the goals of explainable nlp, where the aim is to understand how a system arrived at a certain prediction; these include incompleteness (particularly selectivity), subjectivity, the inclusion of illustrative elements, and the ability of systems to provide explanations even for wrong answers @xcite @xcite . in contrast, in the field of human-computer interaction, human-like explanations are seen more favourably @xcite , indicating tension between the various goals and user groups of self-rationalising systems. the aim of this paper is to systematise properties of human explanations and to gauge to what extent these properties are reflected in the explanations generated by llms. this knowledge can help developers and users of these models understand in which cases the generated explanations are aligned with specific goals, and when a model needs to be adapted or is unfit for the intended use case. to get insights into the properties of llm-generated explanations, we look into the alpaca dataset @xcite . choosing this dataset lets us study the properties of the explanations generated by @xcite , the llm used in the construction of alpaca. however, our findings also have a wider scope, as alpaca is used for ift and the properties it exhibits, including the properties of the explanations contained in it, are likely to be further propagated to derived models fine-tuned on it. ## related work we give a brief overview of the work on self-rationalising models in §2.1. in §2.2 we summarise arguments for and against the use of generated natural language explanations. ## properties of explanations in this section, we introduce the properties of explanations that we will review and discuss in this paper. as llms are largely trained on human-authored text, we expect their generated explanations to be similar to human explanations @xcite . to identify and systematise relevant properties, we take inspiration from work on how humans construct and understand explanations @xcite . in recent years, such work has even targeted the explainable machine learning audience @xcite . it is important to note that human explanations do not all share universal properties. their nature and structure interact heavily with the explanandum, that is, the topic of the explanation. for example, while both an everyday explanation (e.g., why you are late for dinner) and a mathematical proof are human-made explanations, they have little in common @xcite . in this section, we will focus specifically on properties of human explanations that have been pointed out as disadvantageous in the context of explainable nlp, and that we will test for in our experiment. ## experimental setup in this section, we introduce our data and annotation setup. all data, code and ratings can be found at https://github.com/jekunz/ llm-expl-properties.. ## results we separate the results of our human evaluation into two parts: the answers to the first two questions about the existence of explanations in §5.1 and the answers to the latter four questions in §5.2. ## discussion the natural language explanations given by llms are apparently not faithful to the prediction process but the result of the autoregressive pre-training, i.e. they imitate human explanations from the training data, possibly constrained by instruction fine-tuning and other alignment techniques. as such, they exhibit typical properties of human explanations, which we discuss in §6.1. in §6.2 we reflect on our evaluation method and data. finally, in §6.3 we discuss the implications of our findings for different goals of explainable nlp. ## conclusion large language models imitate human explanations in their training data and adopt some of their typical properties. in our analysis of gpt-4 outputs from the alpaca dataset, selectivity and illustrating factors were particularly common. subjectivity was less common, as it was probably mitigated in the alignment and filtering process of gpt-4. misleading explanations were observed rarely, but given that the alpaca dataset is llm-generated, it is likely that the observation will not hold for lower-probability inputs. we discussed the consequences of the presence of these properties and emphasized that it is crucial to consider both the goals and the target groups of the application. for less competent and careful users, there is a risk of shaping false confidence with incomplete, rhetorically convincing but incorrect or biased explanations. however, not all properties that appear undesirable are unequivocally negative: explanations for false predictions may help developers spot the fallacies of the model. unfaithful reasoning can make explanations more accessible with simplifications and illustrative examples. selectivity is often even necessary for generating comprehensible explanations.