Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks (2024)

Abstract: Large language models (LLMs) are proficient at generating fluent text with minimal task-specific supervision. However, their ability to generate rationales for knowledge-intensive tasks (KITs) remains under-explored. Generating rationales for KIT solutions, such as commonsense multiple-choice QA, requires external knowledge to support predictions and refute alternate options. In this work, we consider the task of generating retrieval-augmented rationalization of KIT model predictions via external knowledge guidance within a few-shot setting. Surprisingly, crowd-workers preferred LLM-generated rationales over existing crowd-sourced rationales, generated in a similar knowledge-guided setting, on aspects such as factuality, sufficiency, and convincingness. However, fine-grained evaluation of such rationales highlights the need for further improvements in conciseness, novelty, and domain invariance. Additionally, through an expert-sourced study evaluating the reliability of the rationales, we demonstrate that humans’ trust in LLM-generated rationales erodes when communicated faithfully, i.e., without taking model prediction accuracy into account. We find that even instrumenting simple guardrails can be effective for reliable rationalization.

## introduction in recent years, generating rationales (i.e., freetext explanations) of natural language understanding tasks has been increasingly explored in the field of explainable nlp. such rationales -while less functionally grounded, i.e., they may not entirely reflect the model's behavior -provide an effective interface to interpretably communicate model decisions to end-users @xcite @xcite . generating these rationales via direct supervision @xcite ; narang et al., 2020) or fine-tuning @xcite requires the collection of high-quality human-authored rationales. collecting such rationales via crowd-sourcing is expensive, difficult to standardize, and lacks generalizability to different domains @xcite . recent work @xcite showcases that large language model (llm) generated rationales, obtained via few-shot in-context learning @xcite @xcite , alleviate these challenges while showcasing surprising effectiveness over crowdsourced rationales on dimensions such as human preference. however, characterizing the suitability of llms as rationalizers of knowledge-intensive task (kit) decisions such as commonsense question answering (csqa @xcite ) and open book question answering (obqa @xcite ) requires further investigation due to the difference in scope and setting from prior work @xcite . firstly, kits such as csqa and obqa are framed as multiple-choice questions, requiring models to select one answer from several choices (see figure 1a ). therefore, a corresponding wellformed rationale is required to be (a) comprehensive, i.e., state facts that are not present in the question but are essential for rationalization, and (b) refutation complete, i.e., rationalize why the rest of the choices are incorrect or not best suited as the answer @xcite . we show an example of such a rationale in figure 1b . however, llm-generated rationales in prior work @xcite have only been evaluated on their corroboration capabilities. secondly, llm-generated rationales in prior work are abstractive @xcite , lacking grounding on external knowledge sources crucial for accomplishing the task -kit models designed for csqa and obqa @xcite @xcite refer to external sources such as conceptnet @xcite (see figure 1a ). finally, kit models may predict incorrectly -faithfully rationalizing such mistakes may erode the end-user's trust in the generated rationales. existing approaches in explainable nlp omit the incorrect prediction confounder and evaluate only rationales of correct predictions. however, with llm-generated rationales being increasingly adopted in real-world scenarios, such as rationalizing why a candidate is suitable for an advertised job foot_0 , it is important to scrutinize the practical implications of such deployments and inform guidelines for safe and responsible adoption. given the setting of generating corroborating and refutation complete rationales of kit model decisions, we explore the suitability of retrievalaugmented rationale generation using llms. we enrich the prompt to llms with relevant knowledge retrieved from external sources to condition the rationale generation on facts. more specifically, we generate knowledge-guided rationales containing corroboration and refutation componentssimilar to figure 1b -via few-shot prompting of llms. we conducted three human subject studies to evaluate the effectiveness of such rationales in communicating kit model decisions. the observations from these studies enable coarse-and fine-grained characterization of the strengths and weaknesses of llm-generated knowledge-guided rationalization of kit model decisions. more specifically, we conduct two studies via crowdsourcing to evaluate the preferability and acceptability of such rationales to crowd-workers. in another study involving experts -motivated by existing literature on trust in explainable ai @xcite -we explore the implications of faithfully rationalizing kit model decisions irrespective of their correctness. the crowd-sourced studies demonstrate that, more often than not, crowdworkers prefer llm-generated rationales to crowdsourced rationales in existing datasets, citing their factuality, sufficiency, and convincing refutation. follow-up fine-grained analysis reveals that llm-generated rationales still have significant room for improvement along dimensions such as insightfulness (i.e., providing new information), redundancy (i.e., avoiding repetitive text), and generalizability (i.e., domain invariance.) the expert-sourced study confirms that faithful rationalization of incorrect model predictions degrades humans' trust in the generated rationales. we further explore the utility of instrumenting mechanisms to intervene the incorrect predictions via a reviewthen-rationalize pipeline instead of faithfully rationalizing and find that even simple strategies may help intervene up to 71% of the incorrect predictions. the code and data related to the humansubject studies are publicly available foot_1 . the key contributions of our work include: ## knowledge-enhanced rationalization kit models such as mhgrn @xcite , qagnn @xcite , and dragon (yasunaga et al., 2022) combine language model and knowledge graph representations to solve complex tasks such as commonsense qa @xcite . we aim to generate rationales that corroborate the kit model's prediction with additional relevant facts while refuting the other choices (see figure 1.) our approach is similar to existing retrievalaugmented generation strategies with llms @xcite @xcite . to guide the generation of these rationale components, i.e., corroboration and refutation, we retrieve facts concerning the knowledge-intensive task -e.g., questions and choices in csqa and obqa -from a knowledge graph such as conceptnet @xcite . we then prompt an llm to rationalize the prediction via conditioning on the provided knowledge. figure 2 outlines the rationalization process given an input, i.e., question, choices, and model prediction. given an external knowledge-graph such as con-ceptnet @xcite , we employ the knowledge extraction strategy used in qagnn @xcite first to retrieve the facts relevant to a question and then select top-k (@xmath0. such selection enables us to fit the knowledge facts within the token limits of an llm prompt. we employ greedy decoding-based few-shot prompting to query an llm for rationalization. each example in the prompt contains a qa task, the corresponding kit model prediction, facts retrieved from concept-net, and expert-written rationale corroborating the prediction and refuting other choices. we opted for expert-authored rationales due to their reported effectiveness over crowdsourced rationales @xcite . the paper's authors collaboratively crafted high-quality rationales to compile the expert-written pool. figure 3 outlines the fewshot prompt structure. give a new multiple-choice question; we combine the question, the model prediction, and the corresponding extracted facts with the few-shot examples sampled from the expertpool to formulate the final prompt (see figure 2 .) we provide a detailed description of the prompt design in appendix a ( ## evaluation of rationales due to a lack of suitable automated methods for evaluating the rationale quality @xcite and credibility, we conducted three studies to address the following questions: rq1. how effective are the llm-generated rationales in communicating kit model decisions compared to crowdsourced rationales? ( § 4) rq2. to what degree do the fine-grained rationale characteristics influence its effectiveness and how generalizable are these observations? ( § 5) rq3. how does faithful rationalization of model predictions impact humans' trust in the llmgenerated rationales? ( § 6) datasets and prompts. we select qagnn @xcite as the kit model due to its well-documented code repository and availability of pre-trained model weights. we consider two datasets of multiple-choice qa tasks related to commonsense knowledge, csqa @xcite , and elementary-level science, obqa @xcite . following the existing kit models, we use conceptnet @xcite as our external knowledge source. for both datasets, we report results on a fixed, randomly-sampled 250instance test set. we sample these instances from the test set prepared for these datasets @xcite . we employed gpt-3.5 text-davinci-003 (@xmath1. we randomly selected 40 instances from each of the csqa and obqa datasets -different from the 250 test instances -to be included in the expertwritten example pool. see appendix a for details. ## llms vs humans as rationalizers we first compare llm-generated rationales of the csqa @xcite tasks with corresponding crowdsourced rationales from ecqa dataset @xcite . the ecqa rationales are similar in construct to our setting containing corroboration and refutation of csqa tasks. we exclude @xcite , another crowdsourced free-text rationales dataset, as those rationales are not refutation complete. moreover, ecqa rationales are reported to be overall better than cos-e in rationalizing kit decisions @xcite . we explain the dataset selection criteria in further detail in appendix a. following are the key takeaways: ## acceptability of llm rationalization while pairwise evaluations of preferences provided perspective on the relative quality of the rationales, we conducted another study to independently measure the acceptability of the llm-generated rationales and collect absolute crowd-worker judgments across several aspects related to rationale quality. we evaluated rationales for both csqa and obqa dataset tasks to understand how generalizable these observations are. the key takeaways from the study are as follows: ## towards credible rationalization in the earlier studies, similar to existing work @xcite @xcite , we evaluate llm-generated rationales for cases where model prediction matches the ground truth. we now investigate the implications of rationalization without accounting for model errors, i.e., faithful rationalization, and potential intervention strategies. following are the key highlights of the study: ## related work free-text rationale generation. existing works highlight the effectiveness of free-text rationales in justifying a model's decision to humans in vision @xcite ) and text domains @xcite @xcite . due to cost and generalizability implications of supervised rationale generation, we employ few-shot prompting to elicit rationales from llms following existing work @xcite . both these approaches generate abstractive, corrobora-tive, and faithful rationales. in contrast, we explore the generation of knowledge-guided, corroborative, refutation-complete, and credible rationales. rationale evaluation. existing work employs automatic and human-subject study-based approaches to evaluating rationales. the automated evaluation mechanisms focus on specific aspects of a rationale such as conciseness @xcite , supportiveness @xcite , robustness @xcite , and insightfulness @xcite . human-subject studies @xcite @xcite measure subjective characteristics, such as convincingness and objective properties, such as conciseness, of a rationale. since the effectiveness of free-text rationales lies in natural languagebased seamless communication to end-users, in this work, we prioritized characterizing llm-generated rationales of model decisions communicated to endusers. therefore, we opted for human-subject studies that aim to scrutinize the utility of such rationales for knowledge-intensive tasks, characterize their strengths and limitations, and inform guidelines for safe and responsible adoption. ## conclusion we evaluate llms' capacity to generate effective rationales for knowledge-intensive tasks in a fewshot knowledge-guided setting. we additionally investigate the implications of employing llms as rationalizers of an imperfect model and highlight the negative impact on users' trust. observations from our studies highlight room for improvement in aspects such as task and domain invariant rationalization and robust intervention strategies for real-world usage.