Calibrating LLM s with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring (2024)

Abstract: Generating rationales that justify scoring decisions has been a promising way to facilitate explainability in automated scoring systems. However, existing methods do not match the accuracy of classifier-based methods. Plus, the generated rationales often contain hallucinated information. To address these issues, we propose a novel framework capable of generating more faithful rationales and, more importantly, matching performance with classifier-based black-box scoring systems. We first mimic the human assessment process by querying Large Language Models (LLMs) to generate a thought tree. We then summarise intermediate assessment decisions from each thought tree path for creating synthetic rationale data and rationale preference data. Finally, we utilise the generated synthetic data to calibrate LLMs through a two-step training process: supervised fine-tuning and preference optimization. Extensive experimental results demonstrate that our framework achieves a 38% assessment performance improvement in the QWK score compared to prior work while producing higher-quality rationales, as recognised by human evaluators and LLMs. Our work sheds light on the effectiveness of performing preference optimization using synthetic preference data obtained from thought tree paths. Data and code are available at: https://github.com/lijiazheng99/thought_tree_assessment.

## introduction student answer marking requires substantial human efforts. the human scoring process (e.g., figure 1(a) ) typically involves first understanding the content of the student response, then identifying valid key answer elements (e.g., "student should answer what materials to test in their response"), and finally assigning a score according to a marking rubric (e.g., "2 points for accurately describe two additional pieces of information") @xcite . numerous studies (alikaniotis et al. @xcite have aimed to automate such an assessment pipeline, thereby reducing labour and standardising assessment criteria. traditional automated assessment methods generally address this problem through text classification @xcite or regression @xcite , with the goal of predicting the score for a given student response (figure 1(b) ). recent studies suggested that large language models (llms) could be leveraged to explain student answer scoring. for example, @xcite utilised chatgpt-generated rationales as noisy supervision signals to train a smaller model for simultaneous student answer scoring and rationale generation. however, their framework does not match the accuracy of classification baselines and suffers from hallucination problems, leading to unfaithful and inconsistent rationales. moreover, it does not support the assessment of key elements in student responses according to the marking rubric. to address the aforementioned issues, we propose a novel framework to imitate the human scor-ing process by dividing the scoring of student answers into two steps (figure 1(c) ). initially, an llm is tasked with comparing each key answer element against a student answer and providing a binary assessment decision ("yes" or "no") indicating its presence or absence. these series of intermediate decisions, combined with prompts, ultimately form an assessment thought path @xcite . subsequently, the assessment outcomes for each key element are aggregated using a marking rubric to derive the final score. this aggregation process can either be executed using explicit arithmetic calculations in python code or directly by the llm. each thought tree paths can be faithfully summarised into synthetic responselevel rationales, facilitating explainable automated student answer scoring. furthermore, the thought tree paths could contain trajectory that leads to correct or incorrect assessments. such positive and negative rationale samples generated by the thought tree can be regarded as preference data, which allows us to conveniently construct a synthetic rationale preference dataset. therefore, to calibrate llms for both accurate assessment of student answers and faithful generation of corresponding rationales, we align llms with our preference dataset using advanced rlhf methods @xcite . specifically, our alignment pipeline contains the following two steps: (1) supervised fine-tuning (sft) on synthetic response-level rationale data, followed by (2) preference optimization @xcite on synthetic preference data. to address data scarcity, we generate synthetic data nearly three times larger than the original dataset. our experimental results demonstrate that our framework enhances the explainability of the assessment process by generating more faithful rationales while matching or exceeding the accuracy of traditional text classification approaches. in summary, our contributions are: (1) we proposed a method for generating more faithful assessment rationales by imitating human assessment processes through thought trees; (2) we developed a technique for generating synthetic preference data by modelling preferences based on the correctness of thought tree paths; (3) we carried out extensive experiments, demonstrating that our framework achieved a 38% improvement in qwk compared to the state-of-the-art assessment rationale generation framework while generating more informative and higher-quality rationales. to the best of our knowledge, this is the first work to perform preference optimization using synthetic preference rationales that are generated via thought tree paths. ## thought tree guided rationale generation framework we propose a thought tree guided rationale generation framework to improve scoring performance and generate higher-quality assessment rationales. an overview of our proposed three-stage framework is provided in figure 2 . in this section, we will first introduce the problem setup and then explain the method used in each stage. problem setup datasets for developing automated answer scoring systems typically contain information about questions, such as question prompts, key answer elements, and marking rubrics, along with a set of student responses and the corresponding marks. formally, let @xmath0, y i )} n @xmath1, where x i denotes a student response and y i represents the corresponding mark. let @xmath2, where m is the total number of distinct key answer elements. for a points-based marking scheme, the marking rubric can be viewed as a question-dependent, piece-wise scoring function, f r (•), defined as: ## related work automated student answer scoring existing automated student answer scoring (asas) systems typically frame the problem as a text classification task @xcite . most asas research focuses on essay scoring, evaluating essays based on language quality @xcite @xcite . various approaches have aimed to enhance the trustworthiness of these systems by explaining the decision-making processes through feature analysis @xcite @xcite or visualizing weights and self-attention mechanisms @xcite . however, due to limited resources, the scoring and explanation of science questions have not been extensively studied. @xcite proposed a rationale generation framework for science question scoring, but their approach suffers from hallucination issues and an apparent trade-off in assessment performance. unlike previous works, our framework managed to generate more accurate assessment rationales that are more faithful to the assessment material. intermediate rationale for llm problem solving recent developments in llms have demonstrated advances in reasoning capabilities @xcite @xcite @xcite . however, complex reasoning in a single step remains challenging for llms @xcite . researchers proposed cot @xcite , which aimed at breaking down complex reasoning into intermediate steps. building on this idea, various prompting methods have been proposed to improve the success rate in solving complex tasks, such as self-consistency @xcite , which utilizes the notion of bagging, and tree-of-thought (tot) @xcite , which increases the breadth of search on top of cot. in our framework, we propose a thought tree to imitate the human assessment process via breaking down assessment task into intermediate assessment decisions. our thought tree differs from the tot in two significant ways: (1) our task requires precise planning. instead of prompting llms to generate plans, prompts in our intermediate steps are determined by each key element. consequently, our tree has a fixed maximum depth of m + 1 and width of 2 (m -1) for m key answer elements. (2) the goal of the tot is to find the most promising states that solve the problem more efficiently, whereas our goal is to sample all available paths, regardless of correctness, and generate more data for training. ## conclusion in this work, we introduced a novel thought tree guided rationale generation framework on providing more accurate scoring decisions justified by more faithful rationales. we employed a thought tree to imitate human assessment decision-making to enable more faithful intermediate assessment. each tree path was then summarized into synthetic rationales. extensive experiments, llm evaluation, and human evaluation results showed that calibrating llms with our framework-generated synthetic response-level rationales and synthetic rationale preference data can effectively provide more informative rationales that align with student responses while achieving higher scoring performance. our work highlights the potential for preference optimization using synthetic preference data obtained from thought tree paths.