MGR : Multi-generator Based Rationalization (2023)

Abstract: Rationalization is to employ a generator and a predictor to construct a self-explaining NLP model in which the generator selects a subset of human-intelligible pieces of the input text to the following predictor. However, rationalization suffers from two key challenges, i.e., spurious correlation and degeneration, where the predictor overfits the spurious or meaningless pieces solely selected by the not-yet well-trained generator and in turn deteriorates the generator. Although many studies have been proposed to address the two challenges, they are usually designed separately and do not take both of them into account. In this paper, we propose a simple yet effective method named MGR to simultaneously solve the two problems. The key idea of MGR is to employ multiple generators such that the occurrence stability of real pieces is improved and more meaningful pieces are delivered to the predictor. Empirically, we show that MGR improves the F1 score by up to 20.9% as compared to state-of-the-art methods.

## introduction the widespread use of deep learning in nlp models has led to increased concerns about interpretability. to solve this problem, @xcite proposed rationalization framework rnp in which a generator selects human-intelligible subsets (i.e., rationales) from the input text and feeds them to the subsequent predictor that maximizes the text classification accuracy, as shown in figure 1 . unlike post-hoc approaches for explaining black-box models, the rnp framework has the built-in selfexplaining ability through a cooperative game between the generator and the predictor. rnp and its variants have become one of the mainstreams ## problem definition notation. f g (⋅) and f p (⋅) represent the generator and predictor, respectively. θ g and θ p represent the parameters of the generator and predictor, respectively. d represents the distribution of dataset. we consider the classification problem, where the input is a text sequence @xmath0, x 2 , ⋯, x l ∥ with x i being the i-th token and l being the number of tokens. the label of x is a one-hot vector @xmath1, 1} c , where c is the number of categories. cooperative rationalization. rationalization framework consists of a generator and a predictor. the goal of the generator is to find the most informative pieces containing several tokens in the original input text x. for each sample (x, y ) ∼ d, the generator firstly outputs a sequence of binary mask @xmath2, ⋯, m l ∥ ∈ {0, 1} l . then, it forms the rationale z by the element-wise product of the architecture of mgr. x is the input full text. z i is the selected rationale and ŷi is the predictor's output corresponding to z i . to simplify the notation, we denote @xmath3. in cooperative rationalization, the informativeness of the rationale z provided by the generator is measured by the negative cross entropy -h(y, ŷz ), where ŷz is the output of the predictor with the input being z. consequently, the generator and the predictor are usually optimized cooperatively: regularizer of shortness and coherence. to make the selected rationales human-intelligible, the original rnp constrains the rationales by short and coherent regularization terms. in this paper, we use the constraints updated by @xcite : the first term encourages that the percentage of the tokens being selected as rationales is close to a pre-defined level s. the second term encourages the rationales to be coherent. ## conclusion and future work in this paper, we design a new framework mgr to simultaneously tackle the two major challenges including the spurious correlation and degeneration in the self-explaining rationalization schemes. specifically, we propose leveraging multiple generators to select rationales such that the predictor can have access to more meaningful rationales stably. we theoretically show that the proposed method can solve the two problems. finally, empirical results conducted on various datasets demonstrate the great effectiveness of our proposed method.