Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-text Rationales (2023)

Abstract: Among the remarkable emergent capabilities of large language models (LMs) is free-text rationalization; beyond certain scale, large LMs are capable of generating seemingly useful rationalizations, which in turn, can dramatically enhance their performances on leaderboards. This phenomenon raises a question: can machine generated rationales also be useful for humans, especially when lay humans try to answer questions based on those machine rationales? We observe that human utility of existing rationales is far from satisfactory and expensive to estimate with human studies. Existing metrics like task performance of the LM generating the rationales or similarity between generated and gold rationales are not good indicators of their human utility. While we observe that certain properties of rationales like conciseness and novelty are correlated with their human utility, estimating them without human involvement is challenging. We show that, by estimating a rationale’s helpfulness in answering similar unseen instances, we can measure its human utility to a better extent. We also translate this finding into an automated score, Gen-U, that we propose, which can help improve LMs’ ability to generate rationales with better human utility, while maintaining most of its task performance. Lastly, we release all code and collected data with this project.

## introduction in recent years, there has been a surge of interest in using language models (lms) for human-ai collaboration @xcite . for example, lms have played a large role in reducing human effort for dataset creation @xcite @xcite and helping humans critique text @xcite figure 1 : an illustration of human utility of rationales: here, we show chains of thought (rationales) generated by gpt-3 in two scenarios. the first one is providing knowledge to the human to be able to answer the question, but the second rationale is not useful, and is in fact, misleading the human to answer incorrectly. @xcite . however, the opaque reasoning processes of these lms pose serious concerns about their role in high-stakes decision-making @xcite . recently, many works have explored using lms to generate fluent, human-like free-text rationales foot_0 via natural language @xcite ) that can explain their decisions. further, rationales can reference things beyond the task input, and also support high flexibility in content, style, and length @xcite @xcite . however, evaluating if a rationale of a task-instance contains enough knowledge to help lay humans understand and solve that instance correctly is still under-explored. prior literature for human-ai collaboration has studied plausibility @xcite . however, plausibility only aims to capture human judgement of the rationale supporting lm's predicted label. there has been little work done on evaluating actionable advantages offered by rationales to lay humans in understanding a task, despite the promise of human-ai collaboration @xcite . studying human utility of rationales is important to not only situate them in real-world use cases beyond the involvement of researchers, but also to bridge the gap between human and ai understanding, specifically in scenarios where ai systems perform better. in this work, we shift the paradigm of rationale evaluation, by investigating human utility of rationales in helping lay humans understand and solve a given task correctly. in our study, we observe that human utility of current lms is far from satisfactory (including large lms like gpt-3), with only 20% of generated rationales being actually useful ( §2). given that human evaluations are expensive, we should find a reliable way to measure human utility. we examine the correlation of two straightforward measures like lm task performance and alignment with gold rationales, with human utility and find no usable insights. we also ask humans to evaluate rationales w.r.t eight granular-level properties. while we observe that six out of these eight properties are correlated with human utility, reliably estimating them without human evaluation is still an open question @xcite . in addition to the above observation, we find that high-utility rationales effectively transfer knowledge to humans for solving new instances. ( §3) we create new instances (e.g., questions) by either paraphrasing the original instance in a nontrivial manner (rephrase), editing the original instance so that its correct label is changed (counterfactual), or writing an instance that requires a similar reasoning process as the original instance (similar reasoning). we observe that useful rationales help humans generalize better to new instances, whereas non-helpful rationales even mislead them to answer incorrectly. to follow up on the above finding, we show that we can improve an lm's ability to generate rationales with better human utility. ( §4) we translate this finding into an automated score, gen-u, that reflects the ability of a rationale to help an lm answer generalization instances, that better correlates with human utility (when compared to other metrics like lms' task accuracy). we use gen-u as a reward @xcite while generating rationales and observe that the updated lm generates 2% more useful rationales and gets rid of 4% misleading rationales than before, via human-subject evaluations, without hindering the lms' task performance. this paper presents the first comprehensive study of lay human utility of free-text rationales. by introducing lay humans in the rationale evaluation pipeline, and using their insights into existing lms, we believe our findings can help guide future work on developing methods for efficient and reliable human-ai collaboration. ## human utility of lm-generated rationales we begin by defining human utility, intuitively and formally, and describing the lms that we use for the rest of the paper. based on this definition, we conduct human studies to investigate whether existing lms are capable of generating useful rationales. finally, we follow this up by identifying granularlevel syntactic and semantic properties of rationales can indicate their human utility. ## measuring rationale utility by answering generalization questions as defined in §2, human utility of rationales is determined by their ability to guide humans to correctly solve the task (instances). we follow this up by investigating if humans can generalize to syntactic or semantic perturbations of the original question, while being shown rationales of the original question. this will help us understand if human utility of rationales can also indicate whether rationales help with knowledge transfer for unseen instances. for all our experiments, we use the strat-egyqa dataset. types of generalization questions. for our study, we consider three distinct types of generalization setups. firstly, we evaluate the human h's ability to generalize to non-trivial rephrases of the original question. we avoid simple rephrases like changing a preposition, or removing an adverb so as to avoid near duplicates of the original question. next, we look at counterfactual questions. these questions follow the same reasoning steps as the original question, however, they flip the answer of the original question. lastly, we test h's ability to understand questions that follow a similar reasoning process as the original question, but are not related to the original question. these questions can entail entity swaps, or questions that use one of the reasoning steps to answer the original question. examples of each type of generalization question is shown in ## improving human utility of self-rationalising lms smaller lms like t5-large have better task accuracy, but lack in generating more useful rationales. it can be observed ( §2) that the task performance of a self-rationalizing lm and the human utility of its corresponding generated rationales are not correlated. based on our insights about how useful rationales can help humans generalization to unseen questions, we propose gen-u, which simulates a human through an lm: we define and use gen-u to improve human utility of smaller lms like t5-large, while aiming to maintain their task accuracy (figure 5 ). for all our experiments, we use the strategyqa dataset. lm generalization is a better indicator of rationale's human utility. §3 indicated that generalization to unseen but similar questions via rationales of the original question is a reasonable proxy for human utility of rationales. based on this insight, we propose gen-u, which estimates the generalization performance of an lm variant, after and before being shown a rationale generated by a self-rationalizing model. for a given input-output pair x, y, there exist a set of n generalization questions x g , y @xmath0, y g1 ), (x g2 , y g2 ), . . . , (x gn , y gn )} that is created as per §3. let f be a self-rationalising lm as defined in §2, for which we want to estimate the score. let f i be an lm that takes in x g as its input and predicts a set of labels y i g . similarly, f ir be an lm that takes in x g and the rationale r p generated for x by f, and predicts a set of labels y ir g . gen-u for x is defined as: here, mode returns the most frequently occurring value from the set (similar to majority voting in a set). in other words, if a generalization question is answered incorrectly after being shown the rationale, gen-u is -1, otherwise, gen-u calibrates itself w.r.t the answer before being shown the rationale, to accommodate for cases where the question is easy-to-answer or the lm already contains relevant background knowledge. then, we pick the majority vote of the scores (depicted by the mode) for all the generalization questions for a given original question as its score. to validate if gen-u is indeed usable, we calculate correlations between gen-u and human utility of the corresponding rationales. we find that theill's @xmath1.22, which is indicates that gen-u is a better estimate that f's task accuracy or bertscore similarity between generated and gold rationales (refer gen-u as a reward for updating lm. we use the quark @xcite algorithm with gen-u to improve the human utility of rationales generated by f. quark is an rl-inspired training algorithm that uses reward signals as control tokens on the encoder (or decoder) side, to condition the generation of text. for f, we use the same t5-large setup used in §2. for implementing gen-u, we use t5-base lms for f i and f ir , which are both finetuned on the strategyqa dataset. we begin by first finetuning f for 25 epochs with supervised learning on the strategyqa data, after which we continue training with quark. the final f ′ is obtained after finding the best hyperparameter choices based on gen-u scores for the validation set. ## related work evaluating free-text rationales extractive explanations have been used to improve human's understanding of the model @xcite @xcite @xcite or detecting errors in model predictions @xcite . although prior motivation of generating rationales has been primarily to improve task model performance @xcite @xcite , recent works have evaluated rationales in various ways. wiegreffe et al. ( 2022 ) use human acceptability judgements on over-generated rationales by gpt-3 @xcite . they also evaluate the rationales across seven axes like grammar, factuality, etc. @xcite measure benefits of rationales to lms and compared human written rationales with those generated by gpt-3 across two axes: rationales that provide new information over the input, and those that leak the label directly. rationale generation there are two distinct methods of generating free-text rationales. the first way is to fine-tune an encoder-decoder like model, for example, t5 or it's variations like unifiedqa @xcite @xcite . finetuning t5 to generate rationales @xcite entails appending a tag like explain: in the input text, to nudge the lm to generate rationales during prediction. the generated text can either be separated by structured tags like answer:, explanation:, or it can be unstructured, with the answer followed by a because keyword, followed by the rationale. recent methods have also analysed few-shot prompting of t5 with different input-output templates @xcite . another recent approach of generating free-text rationales is via in-context learning @xcite @xcite . a decoderonly model like gpt-3 or its variants @xcite that are pretrained on a larger corpora of world-knowledge are prompted with demonstrations @xcite , wherein each example contains its corresponding explanation. human utility of human rationales several works in psychology and cognitive science detail the role that human rationales play for human understanding. these studies have shown that human rationales are inherently incomplete and do not capture the complete deductive reasoning process. @xcite . these rationales are used to either provide evidence or procedure behind obtaining a given conclusion for a situation @xcite . furthermore, some works have also detailed the utility human rationales have for human understanding. human rationales have shown to help better generalise to unknown circumstances (lombrozo and gwynne, 2014), justify decision-making @xcite , understand relationships between different world entities @xcite , diagnose when something went or might go wrong, as well as explain one off events that are bizarre @xcite . ## conclusion and future work in this work, we study human utility of free-text rationales, by measuring how well lay humans are able to solve tasks with their help. through extensive human evaluations, we show that human utility of rationales generated by current lms is rather unsatisfactory, and existing available measures do not correlate well with it. we find that generalization ability with rationales as context is a good proxy for human utility, and use it as a reward to improve human utility of lms. there are a lot of scopes to improve human utility of self-rationalising lms, where granular-level properties of rationales can be leveraged directly. furthermore, evaluation of human utility on other tasks (like closed-book qa) is something that is also worth looking at, given that human annotators cannot 'guess' answers for these tasks, making it harder for lms and humans alike.