Weak Reward Model Transforms Generative Models into Robust Causal Event Extraction Systems (2024)

Abstract: The inherent ambiguity of cause and effect boundaries poses a challenge in evaluating causal event extraction tasks. Traditional metrics like Exact Match and BertScore poorly reflect model performance, so we trained evaluation models to approximate human evaluation, achieving high agreement. We used them to perform Reinforcement Learning with extraction models to align them with human preference, prioritising semantic understanding. We successfully explored our approach through multiple datasets, including transferring an evaluator trained on one dataset to another as a way to decrease the reliance on human-annotated data. In that vein, we also propose a weak-to-strong supervision method that uses a fraction of the annotated data to train an evaluation model while still achieving high performance in training an RL model.

## introduction causal event extraction is a crucial task in natural language understanding. it involves identifying cause and effect clauses within an event and the relationship between them. an example text along with its causal event annotations from the finegrained causal reasoning (fcr) dataset @xcite is shown in figure 1 . the emergence of powerful generative models leads to a shift from span-based extraction to the generation of structured information @xcite . however, recent studies suggest that chat-gpt @xcite struggles to surpass smaller supervised models @xcite , even when augmented with chain-of-thought (cot) @xcite and few-shot in-context learning (icl) @xcite . we focus on fine-tuning smaller language models using text annotated with causal and effect spans source text the firm's gross margin is set to stabilize as harley refocuses its efforts on more profitable markets, and our base case assumes that it stabilizes around 32% in 2029, helped by a more measured approach to entering new markets. ## related work we will introduce the recent work in causal extraction tasks, reward models for reinforcement learning, weakly-supervised reward models and data augmentation for generative models. ## approximating human evaluation automated metrics for the evaluation of generated text have limitations in aligning with human evaluation. metrics such as f1 score can measure the overlap between the gold standard extraction and model outputs, but fail to recognise the semantic aspects of such comparisons. in causal event extraction, we often have situations where the output is different and has incomplete overlap with the gold standard but is nonetheless correct. automated metrics are unable to deal with these situations since they cannot account for semantic differences, such as when adding or removing words does not change the meaning of an extraction. one way to circumvent this issue is to employ human annotators to evaluate model outputs. while effective, it is expensive and timeconsuming, severely limiting experimentation and the development of new approaches. to address these limitations, we propose to collect human feedback to train an evaluation model for high-quality feedback generation. the goal is to have an automated way to evaluate model outputs that approximates the judgement a human would have made without the time-consuming and expensive aspects of human evaluation. ## causal event extraction with weak reward model in this section, we introduce our reinforcement learning (rl) framework designed to align our generative extractor with human preferences. we also describe our process for training a weakly supervised reward model, which aims to minimise the data needed for train the reward model. ## experiments datasets. we employ three causal extraction datasets: fcr, fincausal and scite. ## conclusion we have explored several evaluation approaches to address the inherent ambiguity of the causal event extraction task. we find that using a generative model to perform extraction performs well, but that evaluation with automated metrics is challenging. our findings demonstrate the ability to faithfully reproduce human evaluation results using a deberta-based classifier trained on human evaluation of extraction outputs. we also apply the evaluator as a reward model to reinforcement learning, further aligning our generative extractor model to human preferences. we explore multiple datasets, showing how our approach can be generalised and employed our trained evaluator in a transfer setting, reducing the need for further annotation of new data. finally, we propose a weak-to-strong approach where we only use a subset of annotated data to train a weakly-supervised evaluator that can match the performance of the fully-trained version.