Digital Socrates: Evaluating LLM s through Explanation Critiques (2024)

Abstract: While LLMs can provide reasoned explanations along with their answers, the nature and quality of those explanations are still poorly understood. In response, our goal is to define a detailed way of characterizing the explanation capabilities of modern models and to create a nuanced, interpretable explanation evaluation tool that can generate such characterizations automatically, without relying on expensive API calls or human annotations. Our approach is to (a) define the new task of explanation critiquing - identifying and categorizing any main flaw in an explanation and providing suggestions to address the flaw, (b) create a sizeable, human-verified dataset for this task, and (c) train an open-source, automatic critique model (called Digital Socrates) using this data. Through quantitative and qualitative analysis, we demonstrate how Digital Socrates is useful for revealing insights about student models by examining their reasoning chains, and how it can provide high-quality, nuanced, automatic evaluation of those model explanations for the first time. Digital Socrates thus fills an important gap in evaluation tools for understanding and improving the explanation behavior of models.

## introduction large language models (llms) have demonstrated promising end-task performance on a range of tasks @xcite . these models, given their text-generation abilities, can also be prompted @xcite or trained @xcite @xcite to externalize their reasoning 1 as a window into their reasoning capabilities. despite promising end-task performance, 1. carbon is stored in coral reefs as calcium carbonate. 2. coal is formed from the remains of ancient plants, which contain carbon. 3. limestone rocks are formed from the accumulation of calcium carbonate, which is a short-term storage for carbon. ## explanation critiquing: task design we formalize the explanation critiquing task in this section. then in the rest of the paper, we describe how we collect a dataset for this task, build a small and high-performing model for this task, and finally apply the model to reveal interesting insights about qa models in an automatic manner. given a question q, along with a modelgenerated explanation e pred and answer a pred , the task involves giving a critique of the modelgenerated explanation, expressed as a 5-tuple: ## dataset of explanation critiques we introduce ds critique bank, foot_2 a critique dataset where each instance comprises a multiple-choice question (together with answer options and correct answer), a model-generated explanation and answer (using a variety of different models), a critique of the model-generated explanation, as well as (any) human annotations collected on that instance. ds critique bank focuses on questions requiring reasoning, in particular science and commonsense reasoning (section 4.1). the explanations are from different models in popular explanation styles (section 4.2). the generation of critiques as well as the process for crowdsourced and expert annotations are described in section 4.3. ## our model: digital socrates in addition to probing gpt-4 (not fine-tuned) as a critique model, we also train two smaller models, ds-7b and ds-13b, 7 to explore whether smaller language models can be fine-tuned to perform the task of explanation critiquing well. ## experiments we performed experiments and analyses to answer three questions: ( 1 ) what types of insights can the explanation critiquing task reveal about student models? (2) how good is gpt-4 at automatic critiquing? (3) how well can a smaller, trained model (thus avoiding expensive api calls) perform automated critiquing? to answer the first question, we analyze the human-annotated training data to identify what these "ideal" critiques can tell us about the student models (section 7.1). to answer the second and third questions (i.e., can we generate such critiques automatically?), we first ran our three critique models (gpt-4, and the trained models ds-7b, ds-13b) to generate critiques of explanations in the dev partition of our dataset (dscb-dev-crowd-anno). we then obtained human annotations on a subset of these, namely the three critiques on student model explanations for 270 dev questions (drawn from all 10 datasets in dev), i.e., 810 annotated critiques (each done by 3 annotators). 8 finally we use this data to evaluate each critique model (sections 7.2 and 7.3). ## results and analysis in this section, we discuss the types of insights we can get from the explanation critiquing task, based on human-annotated critique data from the dscbtrain-crowd-anno partition of ds critique bank. in subsequent sections, we show that this carries over to model-generated critiques as well. we analyze the distribution of explanation scores e sc separately for questions where the student model answers correctly vs incorrectly (figure 2 ). even when a model gets the answer correct, it can give a flawed reasoning chain, ranging from minor flaws (e @xmath0. conversely, when a model is incorrect in its answer, it could still make some valid points (e @xmath1, thus not all incorrect final answers should be viewed the same as a complete lack of understanding required for the question. such trends are seen across models, regardless of model size and type (see figure 8 in appendix e.1). the explanation critiquing task also provides the dimension of the most significant flaw, giving much explainability and interpretability. using gpt-4 generated critiques rated as good quality by crowdworkers (critique score 2 or 3), figure 3 summarizes the overall fractions of different significant flaws in models' reasoning chains. when models answer questions correctly, there can still be cases where the reasoning chain has flaws (13% across all student models) such as incorrect information. when the model is incorrect, there is a broad spectrum of flaws, such as misunderstanding the question or answer options, incorrect information or reasoning, and reasoning inconsistent with the given answer. as a case study, we compare two student models, gpt-3.5 and llama2-70b, on the science domain (figure 4 ). the models have comparable accuracy, with a slightly lower explanation score e sc for the latter, hinting at shallower understanding. the pie charts showing the flaw dimensions provide an efficient snapshot of the models' weaknesses. for incorrect answers, we see that llama2-70b has a higher rate of incorrect information, which could potentially be mitigated by information retrieval or otherwise improving factuality. gpt-3.5 instead out[]= none misunderstanding lack justification incorrect information missing information incorrect reasoning incomplete reasoning inconsistent answer irrelevant figure 3 : the pie charts show distributions of explanation flaws across all student models. even when models get the answer correct, they may still make errors in their reasoning chain @xcite . when models answer incorrectly, explanation critiquing helps in categorizing and diagnosing errors in the reasoning chain (right). figure 4 : gpt-3.5 and llama-2-70b student models achieve comparable acc on science datasets, with the latter having slightly lower e sc . they also show different patterns in their explanations flaws, e.g., in the amount of incorrect information vs inconsistent answer. our explanation critiquing task formulation delivers even more nuance and interpretability by localizing the flaws. f loc pinpoints flawed parts of model-generated explanations, like "ferromagnetic (c) materials are not typically used for paper clips." foot_7 beyond looking at the dimensions of explanation flaws to note that incorrect information is frequent in llama2-70b model's explanations compared to gpt-3.5, localizing the flaws further informs us about topics in which the model has incorrect information. having a list of such flaws for each model could better facilitate finding patterns in models' gaps in knowledge and reasoning. gpt-4 science remember that the state of matter is indeed a physical property that can distinguish between different types of elements. metals are generally solid at room temperature, while nonmetals can be gases, liquids, or solids. gpt-3.5 science remember that metals are excellent conductors of electricity due to their free electrons. llama2-70b science always remember that in multicellular organisms, the hierarchy of complexity begins with the cell, then progresses to tissues, organs, and finally organ systems. llama2-7b science when conducting an experiment, it's crucial to control for as many variables as possible to ensure that any observed differences are due to the factor you're testing (in this case, the type of plant food), not other factors. commonsense always ensure you understand the question and the answer choices. the question is not asking what skylar will definitely do next, but what someone might want skylar to do next. commonsense ensure that your explanation aligns with the answer you have chosen. it's important to be consistent in your reasoning and conclusion. llama2-70b commonsense be careful with interpreting the language used in the question and answer choices. sometimes, phrases may not be literal but idiomatic or figurative. llama2-7b commonsense sensitivity to spice typically means a person can't handle much spice, not that they add more spice to their food. for instance, providing correct information like "ferromagnetic materials, such as iron, nickel, and cobalt, ... in a variety of applications, including holding sheets of paper together." 10 the general feedback ( with explanation critiquing, as informed by the principles of socratic questioning, the nuanced and interpretable critiques offer various components useful as a systematic, rigorous approach for evaluating model-generated explanations. such finegrained information allows for effective uncovering of specific errors associated with models, which can also be used to improve models in the future. ## further analysis on digital socrates having established the validity of the automatic critique models, especially gpt-4 and ds-13b, we use them to analyze all instances in our dev set. ds-13b critiques reveal similar trends on student models compared to gpt-4. figure 6 presents a representative example comparing the flaw dimension pie charts across critique models. ds-13b and gpt-4 models reveal very similar trends in student models' flaw dimensions, while the ds-7b is slightly less diverse in its critiques. appendix e.5 provides the full comparison. further, ds-13b critiques reveal similar trends on student models compared to humanverified critiques. compared to trends identified in ## conclusion despite advances in llms, the nature and quality of their explanations are still poorly understood. to address this, we introduced the task of explanation critiquing which provides quantitative and qualitative insights into llm performance, beyond answer accuracy. our analysis highlights the value of closely examining model-generated explanations for a deeper understanding of models' capabilities. we create ds critique bank, a dataset of 26,478 nuanced and interpretable critiques under the explanation critiquing task formulation, along with crowdsourced and expert annotations. our digital socrates model (ds-13b), is a capable open critique model comparable in performance to gpt-4 and closely aligned with human annotators in the explanation critiquing task. we also showed how digital socrates can, for the first time, provide rich analyses and insights across a range of student models and datasets, without relying on expensive api calls or human annotation. this work fills an important gap in evaluation tools for understanding and improving the explanation behavior of models. release our models for research purposes only.