ER -Test: Evaluating Explanation Regularization Methods for Language Models (2022)

Abstract: By explaining how humans would solve a given task, human rationales can provide strong learning signal for neural language models (NLMs). Explanation regularization (ER) aims to improve NLM generalization by pushing the NLM’s machine rationales (Which input tokens did the NLM focus on?) to align with human rationales (Which input tokens would humans focus on). Though prior works primarily study ER via in-distribution (ID) evaluation, out-of-distribution (OOD) generalization is often more critical in real-world scenarios, yet ER’s effect on OOD generalization has been underexplored.In this paper, we introduce ER-Test, a framework for evaluating ER models’ OOD generalization along three dimensions: unseen datasets, contrast set tests, and functional tests. Using ER-Test, we comprehensively analyze how ER models’ OOD generalization varies with the rationale alignment criterion (loss function), human rationale type (instance-level v/s task-level), number and choice of rationale-annotated instances, and time budget for rationale annotation. Across two tasks and six datasets, we show that ER has little impact on ID performance but yields large OOD performance gains, with the best ER criterion being task-dependent. Also, ER can improve OOD performance even with task-level or few human rationales. Finally, we find that rationale annotation is more time-efficient than label annotation for improving OOD performance. Our results with ER-Test help demonstrate ER’s utility and establish best practices for using ER effectively.

## introduction neural language models (nlms) have achieved state-of-the-art performance on a broad array of natural language processing (nlp) tasks @xcite . even so, nlms' reasoning processes are notoriously opaque (rudin, * equal contribution. 2019; @xcite , which has spurred significant interest in designing algorithms to automatically explain nlm behavior @xcite @xcite @xcite . most of this work has focused on rationale extraction, which explains a nlm's output on a given task instance by highlighting the input tokens that most influenced the output @xcite @xcite @xcite . recent studies have investigated how machine rationales outputted by rationale extraction algorithms can be utilized to improve nlm decisionmaking @xcite . among these prior works, the most common paradigm is explanation regularization (er), which aims to improve nlm by regularizing the nlm to yield machine rationales that align with human rationales (fig. 1 ) @xcite @xcite @xcite @xcite . human rationales can be created by annotating each training instance individually @xcite @xcite or by applying task-level human priors across all training instances @xcite @xcite . though prior works primarily evaluate er models' in-distribution (id) generalization, the results are mixed, and it is unclear when er is actually helpful. furthermore, out-of-distribution (ood) generalization is often more crucial in real-world settings @xcite , yet er's impact on ood generalization has been underexplored @xcite . in particular, due to the lack of unified comparison of different works using er, little is understood about how ood performance is affected by major design choices in building an er pipeline, like the rationale alignment criterion (i.e., loss function), human rationale type (instance-level vs. tasklevel), number and choice of rationale-annotated instances, and time budget for rationale annotation. in light of this, we propose er-test 1 (fig. 2 ), a framework for evaluating er methods' ood generalization via: (1) unseen datasets, (2) contrast set tests, and (3) functional tests. for (1), er-test tests er models' performance on datasets beyond their training distribution. for (2), er-test tests er models' performance on real-world data instances that are semantically perturbed. for (3), er-test tests er models' performance on synthetic data instances created to capture specific linguistic capabilities. using er-test, we study four questions: (a) which rationale alignment criteria are most effective? (b) is er effective with task-level human rationales? (c) how is er affected by the number/choice of rationale-annotated instances? (d) how does er performance vary with the rationale annotation time budget? for two text classification tasks, we show that er has little impact on id performance but yields large gains on ood performance, with the best er criteria being task-dependent (sec. 5.2). furthermore, er can improve ood performance even with distantlysupervised (sec. 5.3) or few (sec. 5.4) human rationales. finally, we find that rationale annotation yields more improvements than label annotation, specifically with limited time-budget for annotating (sec. 5.5). er-test results further show er's utility and establish best practices for using er effectively. ## explanation regularization (er) given an nlm for an nlp task, the goal of er is to improve nlm generalization on the task by 1 code available at https://github.com/ink-usc/er-test . let f be an nlm for m -class text classification. f usually has a bert-style architecture @xcite , consisting of a transformer encoder @xcite followed by a linear layer with softmax classifier. let x @xmath0.g., a sentence) for task instance i. let y i denote f's predicted class for x i . given f, x i , and y i , the goal of rationale extraction is to output machine rationale r @xmath1, such that each 0 ≤ r t @xmath2. let g denote a rationale extractor, such that r @xmath3, x i , y i ). g can also be used to compute machine rationales w.r.t. other classes besides y i (e.g., target class ẏi ). let ri denote the machine rationale for x i w.r.t. ẏi . given ri obtained via g and f, many works have explored er, in which f is regularized such that ri aligns with human rationale ṙi @xcite @xcite . ṙi can either be human-annotated for individual instances, or generated via human-annotated lexicons for a given task. typically, ṙi is a binary vector, where ones and zeros indicate positive (important) and negative (unimportant) tokens, respectively. we formalize the er loss as: l @xmath4, ṙi ), where φ is an er criterion measuring alignment between ri and ṙi . thus, the full learning objective is: @xmath5, where l task is the task loss (e.g., cross-entropy loss) λ @xmath6.e., loss weight) for l er . while there are many choices for φ, it is unclear how φ impacts training and when certain φ should be preferred. also, as a baseline, let f no-er denote an nlm that is trained without er, such that @xmath7. ## er-test existing works primarily evaluate er models via id generalization @xcite @xcite @xcite @xcite , though a small number of works have done auxiliary evaluations of ood generalization @xcite ; kennedy 🐵 nolan is a great director. text: label: pos @deltaairways i @xmath8: label: pos ood id nolan is a great director. text: label: pos nolan is an awful director . sometimes, nolan is a terrible director . neg neg perturbed text nolan is a very good director . text: label: pos vocabulary tests nolan is not a great director . text: label: logic tests neg nloan is a great director . @hjhbkghkv text: label: robustness tests pos rajeev is a great director . text: label: entity tests pos 1. unseen dataset tests 2. contrast set tests 3. functional tests 🎯 🙈 😈 🙂➡ 🐵➡ figure 2: er-test framework. while existing er works focus on id generalization, er-test evaluates er's ood generalization w.r.t. (a) unseen datasets, (b) contrast set tests, and (c) functional tests (sec. 3, a.2.3). let d be an m -class text classification dataset, which we call the id dataset. assume d can be partitioned into training set d train , development set d dev , and test set d test , where d test is the id test set for d. after training f on d train with er, we measure f's id generalization via task performance on d test and f's ood generalization via (1)-(3). ## er-test design choices an er model consists of three key components: rationale alignment criterion, type of human rationales, and instance selection strategy. er-test evaluates the design choices for each component. ## experiments now that er-test lays a foundation for evaluation and design choices, we conduct a systematic study of the er pipeline through four research questions (fig. 3 ). first, we compare different rationale alignment criteria and analyse which performs better for which task. (rq1: sec. 5.2). second, we compare er pipelines with different types of available human rationales: either dense, instance-level vs. distantly-supervised task-level rationales. (rq2: sec. 5.3) third, we look into strategies on how to select instances on which er should be applied, given resource constraints on the number of rationale-annotated samples. (rq3: sec. 5.4). lastly, we investigate whether er is worth doing, given a time-budget to obtain rationale-annotated instances, while comparing it methods without er and the same time-budget to obtain more labelled data. (rq4: sec. 5.5). ## conclusion and future work in this work, we study explanation regularization (er) -aligning machine rationales with human rationales, in detail. we propose er-test, that evaluates er's ood generalization along three pillars -unseen datasets, contrast set tests and functional tests, and uses it to investigate four research questions surrounding the choice of the rationale alignment criterion, type of human rationale, choice of and time taken to obtain rationale-annotation instances. although er shows minor impact on id task performance, improvements on ood datasets is significant. furthermore, er not only works well with dense, instance-level human rationales, but also with distantly supervised task-level rationales. lastly, er is shown to provide benefits even with limited number of rationale-annotated instances, or within time constraints for rationale annotation. in future, we aim to study er as a tool for improving human-in-the-loop (hitl) debugging of nlms. furthermore, currently er-test is only defined for extractive rationales. human feedback for free-text machine rationales is also a promising extension for er-test.