S OFT T OKEN M ATCHING FOR I NTERPRETABLE L OW -R ESOURCE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a model to tackle classiﬁcation tasks in the presence of very little training data.
To this aim, we introduce a novel matching mechanism to focus on elements of the input, by using vectors that represent semantically meaningful concepts for the task at hand.
By leveraging highlighted portions of the training data, a simple, yet effective, error boosting technique guides the learning process.
In practice, it increases the error associated to relevant parts of the input by a given factor.
Results on text classiﬁcation tasks conﬁrm the beneﬁts of the proposed approach in both balanced and unbalanced cases, thus being of practical use when labeling new examples is expensive.
In addition, the model is interpretable, as it allows for human inspection of the learned weights.
I NTRODUCTION
Gathering and labeling data is a task that can be expensive in terms of time, human effort and resources.
When we cannot rely on already available datasets, training a model with acceptable performance on few data points annotated by few annotators, becomes critical in many practical applications.
This is, indeed, especially important when the data is naturally imbalanced and the demands of gathering samples of the minority class are high.
One important domain in which these issues arise is text classiﬁcation, for example hate-speech (Waseem & Hovy, 2016), web spam (Castillo et al., 2007) and abuse detection (Mishra et al., 2018).
One effective approach to overcome the lack of training data is that of Zaidan et al.
(2007), which consists of augmenting the few data available with rationales , i.e., highlighted portions of the input.
Rationales are usually coupled with feature-engineering to be effective in low resource scenarios.
An alternative way to deal with data sparsity, especially in text classiﬁcation tasks, is to use pretrained language models (LMs) that are ﬁne-tuned on a target domain.
While this approach has been tested on hundreds of training points (Devlin et al., 2018; Howard & Ruder, 2018), it is not clear how it behaves in an even scarcer setting, as the vast parameter space of an LM might pose a problem.
Moreover, ﬁne-tuning a model may require a considerable amount of computing power, therefore restricting its applicability.
On the other hand, some embedding-based models represent the input as a weighted average of words (Kalchbrenner et al., 2014; Sheikh et al., 2016), where the weight is given by a parameter called “reference vector”.
However, these models cannot easily incorporate multiple reference vectors, and they are not interpretable since classiﬁcation works on unreadable embedding features.
In this paper, we propose a novel and efﬁcient representation learning model to address the above issues.
The underlying idea is to focus on relevant words in the input while being able to generalise to semantically similar concepts; this is something akin to what a human would do in the presence of data scarcity.
We therefore introduce two techniques that should coexist to reﬂect our intuition.
First of all, the model to focuses on speciﬁc words by computing soft matching probabilities between each word and multiple vectors which represent semantic concepts.
Secondly, we guide the learning process to learn important concepts thanks to an error boosting technique that exploits rationales.
Basically, it encourages the model to reduce the overall error by improving the prediction associated with highlighted words.
Additionally, by direct inspection of model weights it is possible to understand what words it focuses on; in short, the model is interpretable 1 .
Results across a consistent number of baselines and three datasets also indicate a signiﬁcant improvement in performance.
Interestingly, we always outperform ﬁne-tuned models when little training data is available.
Our model can also assist users to train a classiﬁer for a very speciﬁc task.
As an example, consider training an abstract ﬁltering system with rationales provided by the user itself.
The model will then learn to ﬁlter out papers that are not matching the user’s preferences.
The rest of the paper is structured as follows: Section 2 provides an overview of the existing literature, highlighting similar and different ideas; Section 3 formally introduces the problem as well as our model, providing intuition behind our architectural choices; Section 4 details our experiments and shows our ﬁndings, with a thorough ablation study that disentangles the contribution of each part of the model and a use case on interpretability; ﬁnally, Section 5 summarizes our work.
R ELATED W ORKS
There are different ways in which rationales can be used.
Some works generate rationales, while others exploit them to inform the learning process.
The method proposed by Lei et al.
(2016) tackles text classiﬁcation by learning a distribution of rationales given the text and a distribution of the target class given the rationales.
Interestingly, an additional regularization term is added to the loss to produce rationales that are short and coherent.
The model makes use of high-capacity recurrent neural networks (Schuster & Paliwal, 1997), thus it is tested on large amounts of training data to prevent overﬁtting.
This work was later reﬁned by Bastings et al.
(2019), who proposed a probabilistic version of a similar architecture, where a latent model is responsible for the generation of discrete rationales.
The main advantage of predicting discrete rationales is that it is possible to constrain their maximum number per sample, thus effectively controlling sparsity.
However, it usually requires a large amount of data points to be effective.
The ﬁrst to exploit rationales in a low resource scenario were Zaidan et al.
and Zaidan &
Eisner (2008), by means of a rationale-constrained SVM (Cortes & Vapnik, 1995) and a probabilistic model.
Moreover, the latter is realized as a log-linear classiﬁer with heavy use of featureengineering .
On the other hand, when rationales are deﬁned on features rather than on samples, one can use the Generalized Expectation (GE) criteria (Druck et al., 2007; McCallum et al., 2007) to signiﬁcantly improve the performance of classiﬁers.
Rationales can also be incorporated in the loss function (Barrett et al., 2018), where the attention module (Vaswani et al., 2017) on top of an LSTM (Hochreiter & Schmidhuber, 1997) is forced to attend relevant tokens in a document.
This method was not tested on small datasets, possibly because of the aforementioned issues of high capacity models.
A similar approach has been successfully applied by Bao et al.
(2018) to the weak supervision problem.
However, the model assumes one source domain , with supervised labels, to learn an attention generation module that is then applied to the target domain.
In contrast, our method can be built on a given embedding space with minimum supervision.
Apart from incorporating prior knowledge in the form of rationales, one can augment neural networks with: ﬁrst-order logic (Li & Srikumar, 2019;
Hu et al., 2016a); a corpora of regular expressions (Luo et al., 2018); or massive linguistic constraints (Hu et al., 2016b).
While generally powerful and effective, all these methods require domain-speciﬁc expertise to deﬁne the additional features and constraints that are then explicitly incorporated into the network.
In a different manner, the SoPA architecture of Schwartz et al.
(2018) learns to match surface patterns on text through a differentiable version of ﬁnite state machines.
A weighted combination of these patterns is used to classify a document.
Instead, BabbleLabble (BL) (Hancock et al., 2018) is a method for generating weak classiﬁers from natural language explanations when supervision is scarce.
These are then fed to Data Programming (DP) (Ratner et al., 2016), a probabilistic framework, that outputs a ﬁnal score.
On one hand, BL works well because it exploits a domain-speciﬁc grammar to parse explanations; on the other hand, this grammar must be carefully designed by domain experts.
Finally, the Neural Bag Of Words (NBOW) model (Kalchbrenner et al., 2014) takes an average of token embeddings and applies a logistic regression to classify a document.
Its extension, NBOW (Sheikh et al., 2016), computes an importance score for each word by comparing it with a single reference vector that is learned.
Despite the underlying idea being similar, we propose a different mechanism to focus on relevant words.
In the following, we describe the architecture whose only requirements are i) an embedding space of the input, and ii) additional rationales, though the latter is not strictly required.
As we are shall discuss, the model has a strong inductive bias, which is effective when trained with very little data.
Hereinafter, we refer to our new architecture with the name PARCUS, which stands for Pattern
Representations on Continuous Spaces.
T HE PARCUS MODEL
Let us consider a classiﬁcation task in which a very small labelled dataset D L = { ( x 1 , r 1 , y 1 ) , . . .
, ( x L , r L , y L ) } is given, where x i is an input sample, r i represents the rationale information
and y i is the discrete target label.
For the purpose of this paper, an input is a set of tokens x i = { x 1 i , . . .
, x T i i } of arbitrary size T i .
In addition, x ji ∈ R d , where d is the size of an embedding space obtained using a pretrained model.
Finally, we assume that each token in the input has been marked as relevant or not by annotators, i.e., r i = { r 1 i , . . .
, r T i i } ∈ { 0 , 1 } T i .
Intuition When humans are asked to solve a text classiﬁcation problem after seeing few examples, they tend to look for very simple patterns across the dataset, such as speciﬁc words.
Nevertheless, humans are also able to generalise to semantically similar concepts; our goal is to design a model that reﬂects this ability.
For example, assume that the word “excellent” is important for classifying a movie review as positive.
If we were to work in the character space, a straightforward solution would be to match speciﬁc (sub-)strings in the input, an instance of the so-called pattern matching technique.
Clearly, pattern matching cannot generalize to words that have similar meaning, e.g., “outstanding”.
In this work, we transfer the concept of pattern matching into the embedding space, where semantically similar words are assumed to have “close” representations.
We achieve this via a mechanism that outputs a probability of soft matching between a token and a “reference vector”, which is learned together with the classiﬁcation task to capture discriminative “concepts”.
Differently from bag-of-word methods of Section 2, this model easily accommodates multiple reference vectors, hence it can focus on many different concepts that are critical for classiﬁcation.
In order to guide the learning process using the given rationales, it seems sensible to magnify the error for those words that have been marked as relevant by annotators.
Notwithstanding the simplicity of the idea, the underlying challenge is to effectively embed human knowledge into the reference vectors, which are responsible for the soft matching technique.
In other words, the probabilities of soft matching should be highly correlated with the target class.
Finally, we require that classiﬁcation should be done in such a way that a user can explicitly understand which reference vectors are more important for positive or negative prediction.
This last step is obtained by learning a linear combination of the soft matching probabilities.
The next sections describe the proposed model in depth.
First, we show how to compute and combine soft token matching probabilities, and then we introduce the error boosting technique that incorporates rationales in the training process.
It is worth mentioning that both techniques have been designed to coexist, even though the latter is not strictly necessary to train the model.
3. S OFT T OKEN M ATCHING
We now present the core mechanism that implements soft token matching.
Let us deﬁne a set of parameters P = { p 1 , . .
. , p N }
, p k ∈ R d called prototypes , where N is an hyper-parameter of the model.
A prototype plays a similar role as the reference vector in Sheikh et al.
Going back to our movie review example, one p i ∈ P should ideally adapt to be close to the embedding of the word “excellent”.
To learn the N prototypes, we employ the cosine similarity metric.
Cosine similarity can be seen as a way to measure semantic similarity; its co-domain ranges from − 1 , i.e., opposite in meaning, to 1, i.e., same meaning, with 0 indicating uncorrelation.
Ideally, we would like our prototypes to have near-1 similarity with the relevant tokens in the input.
To this aim, we further deﬁne a gate activation function g : [ − 1 , 1] → [0 , 1] that takes the distance between a token x j i and a prototype p k and outputs a probability of soft matching:
[Equation]
where a is an hyper-parameter.
In practice, the closer to 1 the similarity is, the greater the output of this gated activation, and g ( v ) =
1 ⇔ v = 1 .
By choosing a high value of a we strongly penalize tokens that are associated with low similarity scores.
For completeness, Section A.1 depicts g ( v ) for different values of a .
Such a technique is important because it allows PARCUS to focus on
N concepts that are semantically different while fostering interpretability.
Notice that this method from NBOW2 (Sheikh et al., 2016), as we use prototypes to compute per-token features than importance scores.
3. C OMBINING P ROTOTYPES
Equation 1 computes the probability of soft matching between a token and a prototype.
Likewise, because we have N prototypes, we treat all N probabilities as features associated with that token.
We represent these features as φ k ( x ji ) =
g ( d ( x ji , p k ))
∀ k ∈ 1 , . .
. , N .
Now that we have a notion of multiple soft matching probabilities, we can combine them via AND/OR logical functions.
An approximation of such functions can be straightforwardly implemented through the pseudodifferentiable version of min and max (Paszke et al., 2017):
[Equation]
In Section A.2 we propose a fully differentiable version of the above equations, though min and max signiﬁcantly speed up convergence (due to the absence of non-linearities).
3. I NFERENCE
Finally, we need to linearly combine all F features to output a token prediction y ji .
Let us deﬁne an auxiliary function (omitting the argument x ji to make notation less cluttered):
[Equation]
where square brackets denote concatenation
Then, the token prediction is computed as
[Equation]
where W ∈ R F × C is a matrix of parameters (multi-class prediction with C classes’) and b is the (optional) bias.
It is worth noticing that these features are yet another strong inductive bias, and that the linear model is especially necessary to interpret the model, as detailed in Section 4.3.
Figurecombines all steps of the proposed architecture for token classiﬁcation.
Finally, the input prediction is just a sum of the individual y ji
[Equation]
where σ , from now on, represents the softmax activation.
3. R ATIONALE - DRIVEN E RROR B OOSTING
So far, we have not made use of rationales, which are of fundamental importance to guide the learning process.
Intuitively, we would like the prototypes to softly match those tokens that are relevant for prediction .
It follows that a boosting approach (Freund et al., 1999) is not feasible in this scenario, because we want to weight the importance of tokens rather than of whole samples.
Instead, the method we propose is simple and efﬁcient, and it effectively exploits prior information.
The idea is to boost the error associated with speciﬁc tokens’ predictions to encourage the model to focus on them.
To be more precise, at training time only we modify Equation 6 to take into account the prior information as follows
[Equation]
where f : [0 , 1] → R is an arbitrary exponential function of our choice that boosts the error, e.g., f ( x ) =
e x ; we leave the extension to an adaptive version of f for future works.
In terms of learning, f ( r ji ) boosts the gradient of highlighted tokens while leaving unchanged the rest (i.e., if r ji is 0, our f ( r ji ) outputs a multiplicative factor of 1).
Discussion Regularization of the matrix W plays an important role for effective learning.
We use
L1 and L2 regularization terms on W , as done in (Zou & Hastie, 2005), for two main purposes.
First, the L1 term enforces sparsity, which allows us to more easily interpret the importance of different features.
Secondly, L2 limits the magnitude of the weights, hence avoiding over-compensation of low cosine similarity scores.
Consequently, in order to increase one of the soft-matching probability features, the model is encouraged to make changes to the prototypes rather than to the linear weights; in turn, this translates into a particular prototype being “close” to relevant embeddings.
From a mathematical standpoint, we cannot achieve the same result as Equation 6 by means of an additional loss term as done in Lei et al.
(2016), because gradients would be summed and not multiplied.
Moreover, in our experiments we choose to augment ∆( x ji ) with additional information, such as the probability of “opposite” matching: φ ¬ k ( x ji ) =
g ( − d ( x ji , p k ))
∀ k ∈ 1 , . .
. , N .
Speciﬁcally, when φ ¬ k ( x ji ) ≈
the token x ji and p k have cosine similarity equal to -1, hence they are opposite in meaning.
Implementation details PARCUS can be trained by full-gradient descent in an end-to-end fashion, from the prototypes to the linear weights.
We rely on Pytorch (Paszke et al., 2017) to implement our model in few lines of code; this reﬂects the simplicity and strong inductive bias of our approach, which is necessary in the context we consider.
The error boosting technique is applied by the automatic differentiation packages once we implement Equation 7.
We conclude with remarks on the model complexity.
The total number of parameters is Θ( Nd +
FC ) , which is larger than those needed by simpler models such as logistic regression.
Usually, a restricted number of parameters serves to counteract overﬁtting, by limiting the hypotheses space of the model (Vapnik, 1998).
However, this work tackles the problem from a different and novel perspective, as we prevent the prototype weights from freely changing.
Speciﬁcally, prototype weights vary in a way that depends on the given embedding space, because they tend to be close to some token representation.
If this had not been the case, we could have simply used an MLP in place of
P , which does not perform as well as PARCUS in our experiments.
E XPERIMENTS
This section reports the experimental setting as well as our experimental ﬁndings.
We perform an in-depth analysis on our model through ablation studies, in order to clearly separate the contribution of prototypes from the error boosting technique.
Then, we explain what a model can learn by direct inspection of its parameters.
All code to reproduce experiments is publicly available 2 .
4. E XPERIMENTAL S ETTING
Datasets We empirically validate our method on three different datasets.
First, MovieReview (Zaidan et al., 2007) contains balanced positive and negative movie reviews with rationales.
Secondly, we use the highly imbalanced Spouse dataset from Hancock et al.
(2018), where the task is to tell whether two entities in a given piece of news are married or not.
This is a much harder task than standard classiﬁcation, as the same document can appear multiple times with different given entities and background context greatly varies.
Finally, we use the Hatespeech Twitter dataset of
Waseem & Hovy (2016), which contains short and noisy tweets that can belong to the hate-speech or neutral class.
Datasets’ statistics are reported in the appendix for completeness.
We manually provide rationales for 60 randomly chosen positive samples of both Spouse and Hatespeech (this process required approximately 1 hour).
Setup The experimental evaluation was carried out by measuring performances on the given test set while varying the number of data points used for training.
We used balanced train and validation splits for all models, and the validation set is taken as big as the training one to simulate a real scenario.
As for Spouse, we used the given validation set to fairly compare with the results of Hancock et al.
We chose the pre-trained base version of BERT (Devlin et al., 2018) to provide the embedding space to our method and to other neural baselines as well.
We repeated each experiments 10 times with different random train/validation splits; however, different models have been trained and validated on the same data splits, and we report the hyperparameters table in the appendix.
Moreover, to avoid bad initializations of the ﬁnal re-training (for the selected conﬁguration), we average test performances over 3 runs.
The optimized measure is
Accuracy for MovieReview and F1-score for Spouse and Hatespeech, as the former is perfectly balanced.
We optimize the Cross Entropy loss using the Adam optimizer (Kingma & Ba, 2015) for all the baselines we implement.
Methods To have a fair evaluation with respect to the same embedding space, we train a linear model (Linear) and a single-layer MLP that work on token embeddings (MLP), as well as NBOW and NBOW2.
Importantly, we also ﬁnetune BERT on Spouse and MovieReview.
For Spouse, we propose a regular expression that associates speciﬁc sub-strings ( “wife”, “husb”, “marr” and “knot” ) to the positive class; ideally, our model should be able to focus on such words while also generalizing.
Traditional Supervision (TS) is a logistic regression trained on n-gram features, whereas BL-DM stands for the BabbleLabble pipeline tested on 30 random explanations; results for
TS and BL-DM are taken from Hancock et al.
BL-DM can explicitly exploit the relational information of the Spouse dataset, hence it is a strong baseline.
Moreover, we report results of an SVM (Zaidan et al., 2007) and a log-linear model on language features (Zaidan & Eisner, 2008), both of which are speciﬁcally designed to exploit additional rationales.
On Hatespeech, we compare against a Logistic Regression model based on character n-grams (LR-ngrams), as it was shown to reach state of the art performance (Park & Fung, 2017).
Finally, we perform a number of ablation studies to isolate the contribution of different techniques: i) an MLP with the error boosting technique; ii) our method without highlights; ii) our method with no logical features; iii) our method with φ k features only.
4. R ESULTS & D ISCUSSION
Table 1 presents our empirical results for all three datasets, Results conﬁrm that the choice of a strong inductive bias indeed beneﬁts performances in a very low data regime.
On Spouse, our model strongly outperforms other neural baselines and reaches the manually tuned regular expression with just 60 data points (only 30 of them are positive).
Moreover, TS needs ≈ 50x more data to achieve similar performance, whereas 10 datapoints are sufﬁcient to do better than almost all baselines with a training size of 300, a > 30x improvement which does not depend on the chosen embedding space.
We also found that TS performs much worse than our linear baseline (hence the need for a fair comparison on the embedding space).
With 300 datapoints, our model without highlights has an average
[Table]
F1 score very close to that of BL-DM, which relies on a domain-speciﬁc grammar and parser.
In addition, we note that the reported result (46.5) is not averaged over multiple runs; as a matter of fact, one of our random splits achieves a test score of 46.3, indicating the need for robust evaluation when it comes to experiments on few examples.
Overall, we found that the proposed approach can be really helpful when data is greatly imbalanced, and outperforms models like BERT that are deemed quite performing when ﬁne-tuned on relatively small datasets (Devlin et al., 2018; Howard & Ruder, 2018).
Similar arguments apply to MovieReview, where our model strongly improves over the baselines.
Interestingly, our simple representation learning approach is able to beat the state of the art by a large margin when few data points are available.
Here, NBOW and NBOW2 models proved to be strong baselines, as the mean representation of a document seems to work well.
Generally speaking, the gap between performances is more evident as training size is very scarce, even when compared to other baselines that use rationales.
In light of these results, we performed ablation studies on both datasets to understand if the improvements are only due to prototypes, rationales or both.
Overall, we observe that the strong inductive bias represented by prototypes provides a consistent improvement with respect to the other models, which is especially evident on the Spouse dataset.
Interestingly, the MLP does not beneﬁt from error boosting, which might be explained by the fact that its larger hypotheses space, i.e., unconstrained weights, makes it difﬁcult to diminish the contributions of non-relevant tokens.
Because rationales guide the learning process, their are more important in the extremely low resource scenario, but their effect slowly fades as the training size increases; contrarily to our expectations, PARCUS performed even better on larger amounts of training points without rationales.
On Hatespeech PARCUS still performs fairly well on average, but with 200 traning examples it cannot keep up with the logistic regressor of Park & Fung (2017).
We found two reasons for such behavior: (i) BERT’s tokenizer is unable to accurately splitting tweets, due to their noisy nature, and (ii) character n-grams are strong discriminative features for this task (Nobata et al., 2016), and
BERT does not use them.
To simultaneously solve both issues we switched to Fasttext embeddings (Bojanowski et al., 2017) which are also trained on character n-grams and do not need an additional tokenizer.
Surprisingly, we observed a signiﬁcant improvement on 200 data points, which reduces the gap in performance with the n-grams based model.
Therefore, we conclude that choosing the “right” embedding space can make the difference, which does not necessarily mean using one of the latest and most powerful language models available.
4. P ROVIDING E XPLANATIONS
In this Section we show that PARCUS is interpretable.
To this aim, we train a model using N = prototypes on 60 examples taken from the Spouse dataset.
Then we run the model on unseen data and we inspect the outputs associated with each token.
We then rank them to see what are the most important ones, and we observe that the tokens with highest rank correspond to semantic concepts that are relevant for the task.
Indeed, the model learned to focus on words related to marriage, as well as syntactic variations associated with similar semantics.
Moreover, some of the words were not given as part of rationales in the training set.
The next step is to show that rationales have effectively been incorporated in the prototypes, and how the features of Eq.
4 have been weighted.
We started by inspecting the magnitude of the linear weights W ∈ R 8 × 2 ; speciﬁcally, if the i -th feature is discriminative for a class c , then the i -th row of W will have the c -th element clearly larger than the others.
In our example, we found that φ 1 was important for positive predictions, whereas the other features did not contribute to a particular class.
Therefore, we performed topcosine similarity ranking between tokens and the prototype p 1 .
From the most similar to the least one, we obtained: husband ; marriage ; marrying ; wife ; married ; marry ; ﬁance ; wedding ; ﬁancee ; and girlfriend .
Interestingly, PARCUS has automatically learned to match concepts similar to those provided in natural language form by BabbleLabble Hancock et al.
C ONCLUSIONS
We presented a new methodology to perform classiﬁcation in the low data regime.
We coupled soft token matching with error boosting to focus on concepts that are important for the task at hand.
The model is able to outperform other competitors, including ﬁne-tuned complex models.
Moreover, we showed with a practical example how humans can interpret the predictions in terms of concepts matching.
In conclusion, our model proved to be a useful in tasks where gathering data is challenging.
A
G ATING ACTIVATION FUNCTION
The gating activation function of Eq.
1 controls how similar an embedding and a prototype have to be in order to signiﬁcantly contribute as a feature.
Figure 3 shows different curves for different values of a .
A bigger a squashes low cosine similarities more, as is the case for a = 100 , therefore acting as a stricter ﬁlter.
For completeness, Figure 5 shows the effect of a g( · ) in R 2 , using a = 100 .
[Figure]
A. D IFFERENTIABLE IMPLEMENTATION OF AND AND OR
[Figure]
Given the set { φ 1 , , φ N } ∈ [0 , 1] N , interpreted as N independent “soft matching probabilities”
[Equation]
we would like to compute their joint probability as well as the probability of a single matching out of all possible ones.
In our experiments
[Equation]
we found a pseudo-differentiable implementation of min and max (Eq. 2 and Eq. 3) to speed up convergence.
Alternatively, we tried a fully differentiable version of those two functions as well as the probability of mutually exclusive events, i
[Equation]
e., XOR.
We present the equations for N = 2 , although this can be easily generalized to arbitrary N :
[Equation]
where s controls how squashed is the curve is.
Figure 5 depicts all three curves for s =
[Equation]
A. D ATASETS ’ STATISTICS
In Table 2 we report the statistics of the datasets we used.
When preprocessing the data with BERT, we used a maximum sentence length of 128 for token-based methods, and 512 when ﬁnetuning
BERT.
[Table]
As mentioned in Section 4, each sample of the Spouse dataset contains a pair of entities as well as the sentence.
Therefore, the MLP baseline, ablation studies and our method will make use of an input-mask (at test time only), which reﬂects the fact that we are not interested in those sentences where both our entities of interest do not appear.
It is worth noticing that such relational information should be naturally exploited by methods like BabbleLabble, which rely on domain-speciﬁc grammars.
While this method should help to improve precision, in practice it did not signiﬁcantly affect performances.
A. H YPER - PARAMETERS
Hyper-parameters are used to perform model selection, which returns the best conﬁguration for a given validation split.
We use such conﬁguration to train a model on the whole training set and then evaluate its generalization performances on the unseen test set.
When Data Programming is used, we simply combine model selection of our model with that of data programming.
Since the experiment is repeated 10 times to avoid lucky/unlucky data splits, model selection is performed 10 times as well.
Model assessment, i.e., the measure of performance of our family of models, is evaluated by averaging the 10 different performances on the test set.
When ﬁne-tuning BERT, we mainly follow the guidelines reported in Devlin et al.
[Table]
A. G RADIENT BOOSTING EFFECT ON BACKPROPAGATION
For a loss (cid:96) deﬁned on top of Equation 6 and a true label ˆ y , backpropagation is computed as (abstracting from the sample index i to simplify notation):
[Equation]
where θ are the parameters of the model.
A. R OBUSTNESS TO N OISE
We investigated how injecting random rationales (i.e., ones) in prior information affects the ﬁnal performances.
Clearly, having a 100% noise corresponds to not having rationales at all, therefore we run a simple experiment (with ﬁxed hyper-parameters) 10 times on Spouse, with the goal to study robustness to noise.
Figure 6 shows the result, where different curves stand for different training sizes; as one would expect, for few data points (blue and red curves) noise has bigger inﬂuence whereas for 150 (orange curve) and 300 (green curve) data points the effect is negligible.
This conﬁrms that there is a trade-off where the impact of rationales has little effect on the ﬁnal performances.
A. V ISUALIZATION OF THE MOST IMPORTANT TOKENS
The importance of tokens in a sentence can be inspected by looking at the prototypes p
i , the weight matrix W and the tokens predictions y ji .
In Figure 7 we show the some of the tokens which are responsible for positive prediction on Spouse unseen data, which is consistent with the results of
Section 4.3.