Human Ratings Do Not Reflect Downstream Utility: A Study of Free-Text Explanations for Model Predictions (2022)

Abstract: Models able to generate free-text rationales that explain their output have been proposed as an important step towards interpretable NLP for “reasoning” tasks such as natural language inference and commonsense question answering. However, the relative merits of different architectures and types of rationales are not well understood and hard to measure. In this paper, we contribute two insights to this line of research: First, we find that models trained on gold explanations learn to rely on these but, in the case of the more challenging question answering data set we use, fail when given generated explanations at test time. However, additional fine-tuning on generated explanations teaches the model to distinguish between reliable and unreliable information in explanations. Second, we compare explanations by a generation-only model to those generated by a self-rationalizing model and find that, while the former score higher in terms of validity, factual correctness, and similarity to gold explanations, they are not more useful for downstream classification. We observe that the self-rationalizing model is prone to hallucination, which is punished by most metrics but may add useful context for the classification step.

## introduction adding free-text explanations to nlp models is appealing as such explanations are easy to understand to human users and can include richer reasoning than methods that assign relevance scores to the input, such as lime @xcite or saliency maps @xcite . therefore, several commonsense reasoning data sets have been enriched with natural language explanations @xcite @xcite . however, there is also significant scepticism, as the association between the model's predictions and its generated explanations is unclear. @xcite note that explanations may seem plausible but do not provide true insight into the model's reasoning, which fits the observation that open-ended generation models are prone to hallucinating unfaithful content @xcite . also, human explanations are not designed to be valid (or even complete) mechanisms leading to a correct prediction @xcite . in this work, we study the effects of different design choices and properties of automatically generated explanations on the predictive performance of rationale-augmented models. to this end, we make targeted modifications to the model architecture and compare with gold-standard explanations. a common architecture for rationale-augmented models is a pipeline that maps the input to a rationale and the rationale to the output (@xmath0; @xmath1. pipeline models are faithful by construction, but inferior in their performance. self-rationalizing models that generate the rationale along with the output (@xmath2, but it is hard to assess the faithfulness of their explanations @xcite . we focus on a lessstudied usage of free-text explanations, a rationaleenriched pipeline mapping the input to the rationale and the input along with the rationale to the output (@xmath3; @xmath4. this architecture was originally proposed by @xcite in their cage (commonsense auto-generated explanations) model. in the taxonomy of @xcite , we are dealing with serial-task reasoning models. while not inherently faithful, as a causal path from input to predicted label remains open, these models allow us to study interactions between inputs and explanations more directly than self-rationalizing models because they allow for interventions at the explanation level, prior to the classification step. at the same time, @xcite show that the performance is superior to @xmath5, particularly when annotators are not instructed to provide self-contained explanations. we use the framework of rationale-enriched pipelines to generate insights along two lines: ## background annotating free-form explanations for nlp data sets has gained attention in recent years as language generation models became stronger. the popular natural language inference dataset snli @xcite has been enriched with crowd-sourced text explanations, resulting in e- @xcite . two extensions were created for commonsenseqa @xcite , called @xcite and ecqa @xcite . in semeval-2020 task 4, a subtask was to generate a reason why a natural language statement does not makes sense to humans @xcite . @xcite solve algebraic word problems and generate a series of small steps necessary to derive the answer. textual explanations have also been proposed for self-driving vehicles @xcite . the need for manual annotations of natural language explanations creates challenges, such as annotation costs @xcite . also, human explanations can take various forms and have different goals @xcite and do not necessarily verbalize valid reasoning paths @xcite . ## experimental setup we generate and evaluate explanations in reasoning pipeline models using the following setups: ## results we present our main results together with some additional follow-up experiments. ## discussion we now discuss or results and method. ## conclusion in this paper we compared free-text explanations in variants of a rationale-enriched pipeline: using a single-task versus a self-rationalizing generation model, and training the classifier on gold explanation only versus doing further fine-tuning with generated explanations. an extensive evaluation with similarity-based metrics, utility in downstream classification, and human ratings based on five different criteria shows limitations but also chances of free-text explanations. we see indications that hallucinations occur more frequently in explanations by a self-rationalizing generation model. however, they do not appear to be generally harmful, and may even be useful for downstream predictions in rationale-enriched pipelines if the classification model has the chance to learn to differentiate between reliable and unreliable information. further investigation of hallucinations in rationale-enriched pipelines, e.g. with extractive explanation methods, is an interesting avenue for future research. that human ratings do not reflect classification utility indicates that it is crucial to design annotations and models targeted towards a use case: explanations that convince human raters are not ideal for the goal of performance improvements by providing useful guidance to the model. however, the latter goal is not explicitly accounted for in popular data sets, but the former is not sufficiently met either, as particularly for ecqa, human annotators rate gold explanations low. specialized explanations that maximize one goal at a time would help us understand the differences between human and model "reasoning", and thereby allow us to move towards more faithful free-text explanations.