E xp BERT : Representation Engineering with Natural Language Explanations (2020)

Abstract: Suppose we want to specify the inductive bias that married couples typically go on honeymoons for the task of extracting pairs of spouses from text. In this paper, we allow model developers to specify these types of inductive biases as natural language explanations. We use BERT fine-tuned on MultiNLI to “interpret” these explanations with respect to the input sentence, producing explanation-guided representations of the input. Across three relation extraction tasks, our method, ExpBERT, matches a BERT baseline but with 3–20x less labeled data and improves on the baseline by 3–10 F1 points with the same amount of labeled data.

## introduction consider the relation extraction task of finding spouses in text, and suppose we wanted to specify the inductive bias that married couples typically go on honeymoons. in a traditional feature engineering approach, we might try to construct a "did they go on a honeymoon?" feature and add that to the model. in a modern neural network setting, however, it is not obvious how to use standard approaches like careful neural architecture design or data augmentation to induce such an inductive bias. in a way, while the shift from feature engineering towards end-to-end neural networks and representation learning has alleviated the burden of manual feature engineering and increased model expressivity, it has also reduced our control over the inductive biases of a model. in this paper, we explore using natural language explanations (figure 1 ) to generate features that can augment modern neural representations. this imbues representations with inductive biases corresponding to the explanations, thereby restoring some degree of control while maintaining their expressive power. prior work on training models with explanations use semantic parsers to interpret explanations: the parser converts each explanation into an executable logical form that is executable over the input sentence and uses the resulting outputs as features @xcite or as noisy labels on unlabeled data @xcite . however, semantic parsers can typically only parse low-level statements like "'wife' appears between {o 1 } and {o 2 } and the last word of {o 1 } is the same as the last word of {o 2 }" @xcite . we remove these limitations by using modern distributed language representations, instead of semantic parsers, to interpret language explanations. our approach, expbert (figure 2 ), uses bert @xcite fine-tuned on the multinli natural language inference dataset @xcite to produce features that "interpret" each explanation on an input. we then use these features to augment the input representation. just as a semantic parser grounds an explanation by converting it into a logical form and then executing it, the features produced by bert can be seen as a soft "execution" of the explanation on the input. on three benchmark relation extraction tasks, expbert improves over a bert baseline with no explanations: it achieves an f1 score of 3-10 points higher with the same amount of labeled data, and a similar f1 score as the full-data baseline but with 3-20x less labeled data. expbert also improves on a semantic parsing baseline (+3 to 5 points f1), suggesting that natural language explanations can be richer than low-level, programmatic explanations. ## setup problem. we consider the task of relation extraction: given @xmath0, o 1 , o 2 ), where s is a sequence of words and o 1 and o 2 are two entities that are substrings within s, our goal is to classify the relation @xmath1. the label space y includes a no-relation label if no relation applies. additionally, we are given a set of natural language explanations @xmath2, e 2 , . . . , e n } designed to capture relevant features of the input for classification. these explanations are used to define a global collection of features and are not tied to individual examples. approach. our approach (figure 2 ) uses pretrained neural models to interpret the explanations e in the context of a given input x. formally, we define an interpreter i as any function that takes an input x and explanation e j and produces a feature vector in r d . in our expbert implementation, we choose i to capture whether the explanation e j is entailed by the input x. concretely, we use bert @xcite finetuned on multinli @xcite : we feed wordpiece-tokenized versions of the explanation e j (hypothesis) and the instance x (premise), separated by a [sep] token, to bert. following standard practice, we use the vector at the in addition to v(x), we also map x into an input representation u(x) ∈ r 768|y| by using the same interpreter over textual descriptions of each potential relation. specifically, we map each potential relation y i in the label space y to a textual description r i (figure 2 ), apply i(x, •) to r i , and concatenate the resulting feature vectors: finally, we train a classifier over u(x) and v(x): note that u(x) and v(x) can be obtained in a preprocessing step since i(•, •) is fixed (i.e., we do not additionally fine-tune bert on our tasks). for more model details, please refer to appendix a.1. baselines. we compare expbert against several baselines that train a classifier over the same input representation u(x). noexp trains a classifier only on u(x). the other baselines augment u(x) with variants of the explanation representation v(x). bert+semparser uses the semantic parser from @xcite to convert explanations into executable logical forms. the resulting denotations over the input x (a single bit for each explanation) are used as the explanation representation, i.e., v(x) ∈ {0, 1} n . we use two different sets of explanations for this baseline: our natural language explanations (langexp) and the low-level explanations from @xcite that are more suitable for the semantic parser (progexp). bert+patterns converts explanations into a collection of unigram, bigram, and trigram patterns and creates a binary feature for each pattern based on whether it is contained in s or not. this gives v(x) ∈ {0, 1} n , where n is the number of patterns. finally, we compare expbert against a explanations. to construct explanations, we randomly sampled 50 training examples for each @xmath3. for spouse and disease, we additionally wrote some negative explanations for the no-relation category. to interpret explanations for disease, we use scibert, a variant of bert that is better suited for scientific text @xcite . a list of explanations can be found in appendix a.3. benchmarks. we find that explanations improve model performance across all three datasets: expbert improves on the noexp baseline by +10.6 f1 points on spouse, +2.7 points on disease, and +3.2 points on tacred ( we note that expbert-which uses the full 768-dimensional feature vector from each explanation-outperforms expbert (prob), which summarizes these vectors into one number per explanation, by +2-5 f1 points across all three datasets. data efficiency. collecting a set of explanations e requires additional effort-it took the authors about 1 minute or less to construct each explanation, though we note that it only needs to be done once per dataset (not per example). however, collecting a small number of explanations can significantly and disproportionately reduce the number of labeled examples required. we trained expbert and the noexp baseline with varying fractions of spouse and tacred training data (figure 3 ). expbert matches the noexp baseline with 20x less data on spouse; i.e., we obtain the same performance with expbert with 40 explanations and 2k labeled training examples as with noexp with 22k examples. on tacred, expbert requires 3x less data, obtaining the same performance with 128 explanations and 23k training examples as compared to noexp with 68k examples. these results suggest that the higher-bandwidth signal in language can help models be more dataefficient. ## discussion recent progress in general-purpose language representation models like bert open up new opportunities to incorporate language into learning. in this work, we show how using these models with natural language explanations can allow us to leverage a richer set of explanations than if we were constrained to only use explanations that can be programmatically evaluated, e.g., through ngram matching (bert+patterns) or semantic parsing (bert+semparser). the ability to incorporate prior knowledge of the "right" inductive biases into model representations dangles the prospect of building models that are more robust. however, more work will need to be done to make this approach more broadly applicable. we outline two such avenues of future work. first, combining our expbert approach with more complex state-of-the-art models can be conceptually straightforward (e.g., we could swap out bert-base for a larger model) but can sometimes also require overcoming technical hurdles. for example, we do not fine-tune expbert in this paper; doing so might boost performance, but fine-tuning through all of the explanations on each example is computationally intensive. second, in this paper we provided a proof-ofconcept for several relation extraction tasks, relying on the fact that models trained on existing natural language inference datasets (like multinli) could be applied directly to the input sentence and explanation pair. extending expbert to other natural language tasks where this relationship might not hold is an open problem that would entail finding different ways of interpreting an explanation with respect to the input. the symptoms of {o2} appeared after the administration of {o1} {o2} developed after {o1} patients developed {o2} after being treated with {o1} {o1} contributes indirectly to {o2} {o1} has been associated with the development of {o2} symptoms of {o2} abated after withdrawal of {o1} a greater risk of {o2} was found in the {o1} group compared to a placebo {o2} is a side effect of {o1} {o2} has been reported to occur with {o1} {o2} has been demonstrated after the administration of {o1} {o1} caused the appearance of {o2} use of {o1} can lead to {o2} {o1} can augment {o2} {o1} can increase the risk of {o2} symptoms of {o2} appeared after dosage of {o1} {o1} is a chemical {o2} is a disease {o1} is used for the treatment of {o2} {o1} is known to reduce the symptoms of {o2} {o1} is used for the prevention of {o2} {o1} ameliorates {o2} {o1} induces {o2} {o1} causes a disease other than {o2} {o1} is an organ administering {o1} causes {o2} to worsen {o1} is effective for the treatment of {o2} {o1} has an effect on {o2} {o1} has an attenuating effect on {o2}