Reward Engineering for Generating Semi-structured Explanation (2024)

Abstract: Semi-structured explanation depicts the implicit process of a reasoner with an explicit representation. This explanation highlights how available information in a specific query is utilised and supplemented with information a reasoner produces from its internal weights towards generating an answer. Despite the recent improvements in generative capabilities of language models, producing structured explanations to verify a model’s true reasoning capabilities remains a challenge. This issue is particularly pronounced for not-so-large LMs (e.g., FLAN-T5-XXL). In this work, we first underscore the limitations of supervised fine-tuning (SFT) in tackling this challenge, and then introduce a carefully crafted reward engineering method in reinforcement learning (RL) to better address this problem. We investigate multiple reward aggregation methods and provide a detailed discussion which sheds light on the promising potential of RL for future research. Our proposed method on two semi-structured explanation generation benchmarks (ExplaGraph and COPA-SSE) achieves new state-of-the-art results.

## introduction language models have shown great capability in complex reasoning tasks @xcite @xcite @xcite @xcite . despite their proficiency in generating accurate results, a comprehensive assessment of the models' true capabilities in reaching the correct output necessitates an explainable mechanism. in this spirit, generating structured explanations @xcite is a viable approach as they explicitly representing the relationships between facts employed during reasoning, and are amenable to evaluation. unstructured natural language explanations lack these figure 1 : given the belief and argument, the task is to predict the stance (support/counter) and generate an explanation graph representing the reasoning process. the explanation graph under sft+rl is more expressive. for this purpose, @xcite propose to use multiple models for predicting answer, internal nodes, external nodes and relations. @xcite incorporate a generative pre-training mechanism over synthetic graphs by aligning input pairs of text-graph to improve the model's capability in generating semi-structured explanation. both works train separate models for prediction of response, and generation of explanations. it is reasonable to expect that even a moderately-sized language model such as flan-t5-xxl @xcite should be capable of producing both answers and the corresponding structured explanations. we investigate this in our work. in parallel, large lms at the scale of @xcite have shown a great capability in producing both an answer and an unstructured reasoning trace through methods like chain-of-thought (cot) @xcite . one might hope that an ideal structured representation of the reasoning trace could also be comfortably surfaced via in-context prompting of llms. but it has been demonstrated that llms struggle to generate structured format output @xcite . we empirically verify this struggle in the context of generating structured explanations. our objective is to equip moderately-sized lms with the ability to not only provide answers but also generate structured explanations. to facilitate this, we first utilise supervised fine-tuning (sft) as the de-facto solution. we then turn our focus to rlhf foot_0 as a mechanism to further align the explanations with ground-truth on top of sft. we design a reward engineering method in rl and explore multiple reward aggregations that leverage both reward modelling and reward metrics. our proposed method, implemented on the backbone of a flan-t5 @xcite , achieves new state-of-the-art results on two benchmarks, expla-graph @xcite and copa-sse @xcite . as a byproduct, our empirical comparison also highlight the limitations of llms like gpt-4 and gpt-3.5-instruct to succeed at structured explanation generation (seg). furthermore, we delve into a discussion on rl for seg and highlight what reward metrics work better, and spotlight the challenges (i.e., reward hacking) of balancing the dynamic of policy optimization. we hope the findings of our work to shed light on both challenges and potentials of rl in seg as well as the broader space of graph generation. ## semi-structured explanation structured explanation refers to a specific form of explanation that highlights the underlying decisionmaking processes of the model via an explicit representation of relationships between different reasoning factors. in this section, we briefly review different forms of explanations and introduce the semi-structured explanation tasks of our interest. ## conclusion in this work, we focused on the semi-structured explanation generation task and proposed to train a single model with sft+rl to generate both answers and structured explanations. we highlighted the inadequacy of sft in performing this complex task, and proposed a carefully designed reward engineering method in rl to better address this problem. we investigated different reward aggregation methods and conduct extensive experiments under different settings to better highlight the dynamic of the rl objective function and reward choices. our method achieves the new sota results on two seg benchmarks, explagraph and copa-sse. we provide detailed analysis from different perspectives and hope these empirical findings will be beneficial for the future research on investigating rl in seg. weinberger, and yoav artzi. 2019. bertscore: evaluating text generation with bert. arxiv, abs/1904.09675.