SPECTRA : Sparse Structured Text Rationalization (2021)

Abstract: Selective rationalization aims to produce decisions along with rationales (e.g., text highlights or word alignments between two sentences). Commonly, rationales are modeled as stochastic binary masks, requiring sampling-based gradient estimators, which complicates training and requires careful hyperparameter tuning. Sparse attention mechanisms are a deterministic alternative, but they lack a way to regularize the rationale extraction (e.g., to control the sparsity of a text highlight or the number of alignments). In this paper, we present a unified framework for deterministic extraction of structured explanations via constrained inference on a factor graph, forming a differentiable layer. Our approach greatly eases training and rationale regularization, generally outperforming previous work on what comes to performance and plausibility of the extracted rationales. We further provide a comparative study of stochastic and deterministic methods for rationale extraction for classification and natural language inference tasks, jointly assessing their predictive power, quality of the explanations, and model variability.

## introduction selective rationalization @xcite @xcite ) is a powerful explainability method, in which we construct models (rationalizers) that produce an explanation or rationale (e.g: text highlights or alignments; @xcite ) along with the decision. one, if not the main, drawback of rationalizers is that it is difficult to train the generator and the predictor jointly under instance-level supervision @xcite . hard attention mechanisms that stochastically sample rationales employ regularization to encourage sparsity and contiguity, and make it necessary to estimate gradients using the score function estimator (sfe), also known as re-inforce @xcite , or reparameterized gradients @xcite . both of these factors substantially complicate training by requiring sophisticated hyperparameter tuning and lead to brittle and fragile models that exhibit high variance over multiple runs. other works use strategies such as top-k to map token-level scores to rationales, but also require gradient estimations to train both modules jointly @xcite . in turn, sparse attention mechanisms @xcite are deterministic and have exact gradients, but lack a direct way to control sparsity and contiguity in the rationale extraction. this raises the question: how can we build an easy-to-train fully differentiable rationalizer that allows for flexible constrained rationale extraction? to answer this question, we introduce sparse structured text rationalization (spectra), which employs lp-sparsemap @xcite , a constrained structured prediction algorithm, to provide a deterministic, flexible and modular rationale extraction process. we exploit our method's inherent flexibility to extract highlights and interpretable text matchings with a diverse set of constraints. our contributions are: experiments on selective rationalization for sentiment classification and natural language inference (nli) tasks show that our proposed approach method deterministic training exact gradients constrained extraction encourages contiguity lei et al. (2016) bastings et al. (2019) treviso and martins (2020) spectra (ours) ## deterministic structured rationalizers the idea behind our approach for selective rationalization is very simple: leverage the inherent flexibility and modularity of lp-sparsemap for constrained, deterministic and fully differentiable rationale extraction. ## related work selective rationalization. there is a long string of work on interpreting predictions made by neural networks @xcite @xcite @xcite . our paper focus on selective rationalizers, which have been used for extraction of text highlights @xcite @xcite @xcite ) and text matchings @xcite . most works rely on stochastic rationale generation or deterministic attention mechanisms, but the two approaches have never been extensively compared. our work adds that comparison and contributes with an easy-to-train fully differentiable rationalizer that allows for flexible constrained rationale extraction. our strategy for rationalization based on sparse structured prediction on factor graphs constitutes a unified framework for deterministic extraction of different structured rationales. ## conclusions we have proposed spectra, an easy-to-train fully differentiable rationalizer that allows for flexible constrained rationale extraction. we have provided a comparative study with stochastic and deterministic approaches for rationalization, showing that spectra generally outperforms previous rationalizers in text classification and natural language inference tasks. moreover, it does so while exhibiting less variability than stochastic methods and easing regularization of the rationale extraction when compared to previous deterministic approaches. our framework constitutes a unified framework for deterministic extraction of different structured rationales. we hope that our work spurs future research on rationalization for different structured explanations.