AGR : Reinforced Causal Agent-Guided Self-explaining Rationalization (2024)

Abstract: Most existing rationalization approaches are susceptible to degeneration accumulation due to a lack of effective control over the learning direction of the model during training. To address this issue, we propose a novel approach AGR ( A gent- G uided R ationalization), guiding the next action of the model based on its current training state. Specifically, we introduce causal intervention calculus to quantify the causal effects inherent during rationale training, and utilize reinforcement learning process to refine the learning bias of them. Furthermore, we pretrain an agent within this reinforced causal environment to guide the next step of the model. We theoretically demonstrate that a good model needs the desired guidance, and empirically show the effectiveness of our approach, outperforming existing state-of-the-art methods on BeerAdvocate and HotelReview datasets.

## introduction to explain the prediction of neural networks, selective rationalization task @xcite @xcite has been studied in recent years. as shown in figure 1 , it aims to select a small and human-intelligible subset (i.e., rationale) from the input to support and explain the prediction results when yielding them. as an interpretable diagram, rationalization holds significant potential for elucidating the decision-making process of predictive models, building trust, and deriving insightful and pertinent insights @xcite @xcite . various approaches have been proposed for rationalization, spanning from early rationale sampling-based methods @xcite @xcite to the extra-component-based methods @xcite @xcite @xcite . these methods predominantly concentrate on improving the performance of rationalization models by either refining the sampling directly or aligning additional information beyond the rationale, resulting in impressive results. however, to the best of our knowledge, the current methods are prone to degeneration accumulation foot_0 since they usually do not discern whether the generator during training has produced unmeaningful or flawed rationales; instead, they directly pass them to the predictor even if generated rationales are degraded. for instance, the underlined rationale in figure 1 is degraded, as the word :::::::::: appearance alone does not reliably determine the sentiment polarity of input x. but the predictor overfits to this uninformative rationale and classifies the sentiment according to whether "appearance" is included in the rationale. consequently, when the predictor receives degraded rationales, it steers the model towards an undesirable direction (aka., learning bias). thus, optimizing this bias during training is crucial for ensuring the model's generalization performance. the proposed methods @xcite @xcite fall short in considering rationalization optimization comprehensively, neglecting existing causality during rationale learning. although they often employ causal theory to uncover relationships between rationale pieces, they struggle to directly optimize the cooperative game dynamics between the generator and predictor during training. as shown in figure 1 , optimizing rationale from "appearance" to "appearance: light yellow to almost clear" necessitates evaluating the causal impact on target prediction, guiding the model's subsequent optimization. thus, if we could construct a guiding signal to reward or penalize the learning behavior of the model, this would significantly reduce the model's learning bias during training, alleviating the problem of degeneration accumulation. to address the above problems, we propose a novel rationalization method named agr (agent-guided rationalization), which leverages a reinforced causal agent to guide the cooperative game optimization during rationale training, as shown in figure 2 . in particular, 1) we quantify the causal effects in the rationale optimization process, and design a reinforcement learning (rl) process (e.g., markov decision) to refine the learning bias during training. 2) we further pretrain an agent within reinforced causal environment to guide next actions by a system of rewards. we also theoretically illustrate that a robust model needs the desired guidance. 3) experimental results demonstrate the effectiveness of our approach, surpassing state-of-the-art methods on beeradvocate and hotelreview datasets. ## problem formulation notation. following previous research @xcite , we consider the classification problem and denote the generator and predictor as f g (•) and f p (•), with θ g and θ p representing their parameters. the input text @xmath0, x 2 , ..., x l ](1 ≤ @xmath1, where l is the number of tokens. the label of x is a one-hot vector @xmath2, 1} c , where c is the number of categories. cooperative game for rationalization. the f g (•) selects the most informative pieces from x by a sequence of binary mask @xmath3, ..., m l ] ∈ {0, 1} l . then, it forms the rationale @xmath4, m 2 x 2 , ..., m l x l ], where the informativeness of z is measured by the negative cross entropy -h(y, ŷ ). consequently, the f g (•) and f p (•) are optimized cooperatively by in addition, rationales are usually constrained by compact and coherent regularization terms @xcite , where s is a pre-defined sparsity level. predictor predictor agent agent guide i z i s i m i m generator the i-th update reinforced environment policy network probability distribution action sample i s i s i r guide . . . generate sample act state update reward state: i i @xmath5.05 0.48 0.15 0.12 0.10 . . . reward x the architecture of agr. x and ŷ are the input and output. s i is the i-th update state of rationale, while si is the state after guidance by the agent. ## reinforced causal agent in this section, we present our reinforced causal agent, considering both causal effect and learning bias of degeneration during rationale training. ## agent-guided rationalization as depicted in figure 2 , following the establishment of the environment for the reinforced causal agent, we delineate the construction and training of the policy network q ϕ .