Are Deep Sequence Classifiers Good at Non-Trivial Generalization?
Abstract
Recent advances in deep learning models for sequence classification have greatly improved their classification accuracy, specially when large training sets are available.
However, several works have suggested that under some settings the predictions made by these models are poorly calibrated.
In this work we study binary sequence classification problems and we look at model calibration from a different perspective by asking the question: Are deep learning models capable of learning the underlying target class distribution?
We focus on sparse sequence classification, that is problems in which the target class is rare and compare three deep learning sequence classification models.
We develop an evaluation that measures how well a classifier is learning the target class distribution.
In addition, our evaluation disentangles good performance achieved by mere compression of the training sequences versus performance achieved by proper model generalization.
Our results suggest that in this binary setting the deep-learning models are indeed able to learn the underlying class distribution in a non-trivial manner, i.e. by proper generalization beyond data compression.
Introduction
Recent advances in deep learning models for sequence classification have greatly improved their classification accuracy, specially when large training sets are available.
However, several works have suggested that under some settings the predictions made by these models are poorly calibrated: a model might provide in general correct predictions but give bad estimates of the confidence scores
Pr( y | x ) .
Having well calibrated estimates of Pr( y | x ) is crucial to many machine learning applications.
For example, this is critical in systems in which an end user needs to make a decision based on the prediction of the model.
Here a good estimate is as important as a good prediction.
In this work we study binary sequence classification with a focus on sparse sequence classification tasks in which only a very small fraction of the sequences in the domain belong to the target class.
These types of problems naturally arise in several applications, for example in NLP or in
Computational Biology where distinguishing relevant segments in DNA sequences is challenging as it is postulated that only about 1–3% of segments have any biological significance [Eskin, 2002].
While traditional evaluations on model calibration have focused on the quality of the conditional prediction Pr M ( y | x ) in this paper we look at model evaluation from a different perspective by asking the question: Are deep learning models capables of learning the underlying target distribution?
The main idea is quite simple, if we have access to large quantities of unlabeled data that accurately represent the input distribution we can estimate the joint distribution Pr M ( y, x ) implicitly induced by the conditional model and then we can compare it to the true distribution.
We conduct experiments using three deep-learning models: an RNN a Transformer and a WFA (i.e.
Weighted Finite Automata, a form of RNN with linear activation).
Our results suggest that in the binary classification setting the deep-learning models are indeed able to learn the underlying class distribution in a non-trivial manner.
These results seem to be consistent with previous studies showing that deep-learning models provide calibrated predictions for binary classification problems [NiculescuMizil and Caruana, 2005]
Our main contributions are: 1)
We present a novel evaluation framework for sequence prediction models.
By exploiting unlabeled data , we evaluate the model with respect to the implicitly induced joint distribution.
Our evaluation distinguishes performance over components of the distribution seen in training data as well as unseen components explicitly differentiating compression from generalization.
Our experiments on a sparse sequence classification task show that deep learning architectures are able to induce good distributions in a non-trivial manner.
Related Work
In the recent literature several works have addressed the problem of model calibration.
Some of these studies have shown that for some deep learning architectures, the predictions produced by the models are not well calibrated [Guo et al., 2017]
[Kumar and Sarawagi, 2019] especially for non pre-trained transformers [Desai and Durrett, 2020].
In contrast, for the case of binary classification some previous work has suggested that they are indeed well calibrated [Niculescu-Mizil and Caruana, 2005].
To our knowledge we are the first ones to study calibration by looking at the joint distribution Pr( x, y ) induced by the learned classifier instead of the quality of the conditional class predictions.
Calibration aside, several works have compared the classification performance of different deeplearning architectures: CNNs and RNNs [Józefowicz et al., 2016]
[Yin et al., 2017], Transformers and RNNs [Karita et al., 2019]
[Lakew et al., 2018], Transformers and CNNs [Kolesnikov et al., 2021] [Pinto et al., 2021]
[Bai et al., 2021], WFA to CNNs [Quattoni and Carreras, 2020] and WFA to RNNs [Quattoni and Carreras, 2019].
Finally, an orthogonal related problem is that of developing deep learning models for density estimation.
These include Transformers [Fakoor et al., 2020], Autoregressive Networks [Uria et al., 2016]
[Oliva et al., 2018] and Flow Models [Durkan et al., 2019]
[De Cao et al., 2020].
Evaluating Deep-Learning Model as Moment Predictors
Our goal is to evaluate models defined over sequences of discrete symbols.
More precisely we consider an alphabet Σ and the set of all possible sequences Σ ⋆ .
In general, we can think of probabilistic binary sequence classifiers as functions from Σ ⋆ → [0 , 1] .
In our setting we assume that we have access to a large set of sequences U
= { x (1) , . . .
, x ( u ) } sampled according to the underlying distribution over Σ ⋆ .
We can think of U as a large set of unlabeled sequences that represent the domain of the sequence classification task.
The target class Y that we wish to learn is a subset of sequences in U , that is Y ⊂ U .
We are particularly interested in cases where the target class is rare, that is cases in which | Y | is significantly smaller than | U | .
We will create a labeled training set T = { ( x (1) , y (1) ) , . . .
, ( x ( m ) , y ( m ) ) } of size m by sampling sequences from U and labeling y = 1 as positive instances if they appear in Y , and otherwise as negative instances y = 0 .
Section 4 contains further details about how we create the task data from an existing dataset, for various training sizes m .
Given a training set we will train a sequence classifier M that defines a distribution Pr M ( y | x ) .
In this work, we consider three sequence classification models: an RNN , a Transformer and a WFA .
As recurrent neural network (RNN) we employ a multi-layer LSTM
[Hochreiter and Schmidhuber, 1997] with a binary classification feed-forward layer on top.
For the Transformer we select the BERT [Devlin et al., 2019] architecture, we don’t use the pre-trained weights and we expand the embeddings with new randomly initialized ones to deal with the protein dataset vocabulary.
Both models output the conditional probability Pr( l | x ) where x is a sentence and l a label.
We also evaluate a WFA (Weighted Finite Automata; which in essence is an RNN with linear activation functions [Rabusseau et al., 2019]) employing the ensemble proposed in Quattoni and Carreras [2020].
3.1 Evaluation Metrics
The sequence model M that we wish to evaluate defines a conditional distribution Pr M ( y | x ) .
Generally, we would like to compute the true error of M by evaluating the model on the true distribution
Pr( x, y ) .
In this section, we will use the set U as a proxy for Pr( x ) , and specifically we will estimate the moments of M on the joint distribution Pr M ( x, y ) that is implicitly induced by the model over the target class.
1 More precisely, consider the set:
[Equation]
where z n is a sub-sequence of size n , z n ∈ x indicates that z n is a sub-sequence of a domain sequence x , and therefore Z n is the set of all sub-sequences of size n observed in U .
We use these sub-sequences as the support set to compute moments of a model, and we define the moment function
E M : z n × U → R as:
[Equation]
where z n is a sub-sequence of length n and #[ z n ∈ x ] is a function that counts the number of times that z n appears in sequence x .
From now on, we will assume a fixed U and drop it from the expression, and we will refer to the sub-sequences z n and their values E M ( z n ) as the moments of M .
Since we assume that U is a good representation of the the domain, E M ( z n ) can be regarded as an estimate of the expected number of times that we should observe z n in a sample from Pr M ( x, y ) .
Given sets U and Y , the moments from the true joint distribution Pr Y ( x, y ) can be computed similarly as:
[Equation]
Putting it all together, if we have a domain represented by a set of unlabeled sequences U , a subset of target sequences Y and a model Pr M ( y | x ) , we can compute the model’s moment function E M ( z n ) and compare it to the true moments E Y ( x n ) .
Notice that for a model that learns the target class perfectly (i.e. a model that gives accurate predictions and that it is perfectly calibrated) we will have
E M ( z n ) =
E Y ( z n ) for all moments in N .
Now that we have the necessary functions, we propose to evaluate a model M by comparing E M and
E Y .
To compare the moment functions we propose the following metrics:
MSPC :
This metric measures the Spearman rank correlation between gold and model moments of a fixed length.
The Spearman correlation coefficient measures the strength and direction of association between ranked variables.
In our evaluation a high MSPC means that the model sorts the moments of the distribution in a way that is similar to the gold ordering.
The MSPC is computed as:
[Equation]
where rank[ E ( z n )] are the raw function scores for all z n ∈ Z n converted to ranks.
In essence, this metric measures the agreement on partial orderings induced by the model and gold moment functions
That is, given any two pairs of moments of a fixed length z n and z ′ n whenever E Y ( z n ) > E Y ( z ′ n ) we want E M ( z n ) > E M ( z ′ n ) .
MSPCP : While MSPC is a useful metric when evaluating the class distribution induced by a model
we need to keep in mind that for longer moments E Y ( z n ) will be sparse and for most inputs it will be 0 since we are targeting a rare class.
So in this context we compute the Spearman rank correlation between gold non-zero moments and the model prediction for those moments , this will give an idea of how well the model would sort the moments of the target distribution if it knew the true support of the target moments function.
MR : as MSPC might be overly harsh, a complementary alternative metric in this case is the mean-rank given to the target non-zero moments :
[Equation]
This metric captures how well a model performs in the task of making the binary prediction of which moments should be non-zero for the target class.
We believe that it is important to make a distinction between the ability of a model to compress data versus its ability to generalize to unseen data.
A natural way to achieve this goal is to make a distinction between moments that have been seen in the training set, versus moments that have not been seen.
That is, for any training set T and moment length n we can define a subset Z Tn ⊆
Z n :
[Equation]
Given training set T we can define its analogous unseen evaluation by computing the metric over Z n \ Z Tn .
We call these new metrics: MSPC-U , MSPCP-U and MR-U .
In order to evaluate the generalization ability of a model we will also consider a baseline model that memorizes the training data and makes predictions with the observed training moments .
More precisely, given a training set T the baseline will compute the moments function as:
[Equation]
Experiments
To generate the data used in our experiments we use the PFAM dataset [El-Gebali et al., 2018].
This dataset consists of aminoacid sequences pertaining to different protein families.
Deep-models have been shown to be very successful at classifying the complete sequences [Bileschi et al., 2019]
[Szalkai and Grolmusz, 2018].
To derive a more challenging task we decide to work with segments and we generate the unlabeled set U by sampling sequence segments of size 5.
To create our target class Y we pick a protein-family and select all segments that appear in sequences from that family.
We select as our target the most rare or sparse family based on the ratio | Y | / | U | .
We also select a second family as validation data.
We create 5 labeled training sets T = { ( x (1) , y (1) ) , . . .
, ( x ( m ) , y ( m ) ) } by sampling sequences with m ∈ { 400 , 1000 , 2000 , 4000 , 8000 } .
For the optimization details refer to appendix A.
4.1 Results
To summarize the results, for each training set size and model, we compute an aggregate metric, which we refer as MICRO (MC).
The MICRO score is defined as a weighted average of the scores for moments of different lengths, where the weight is proportional to the total number of test-moments of that size.
The full breakdown of results is available in appendix D.
Figure 1a shows MICRO MSPC for the three models and the baseline.
The first observation is that all models perform significantly better than the baseline.
Recall that the baseline is essentially remembering the moments that it has observed at training, therefore it is natural that as we increase the number of training sequences its performance approaches that of the learned models.
Figure 1b shows the analogous results when MSPC is computed over unseen moments , the baseline in this case is undefined since by definition the expectation over all unseen moments will be zero.
Still the models seem to be learning to rank unseen moments , and therefore they are making proper generalizations.
However, one obvious observation is that the MSPC is not very high in absolute terms (appendix D tables, both in the all and unseen scenarios).
Looking at the absolute value of the MSPC might be misleading because we are modeling a sparse sequence classification task.
In other words, the gold expectation for most of the moments in the domain is zero and in this case the MSPC metric might be too harsh.
Thus, as described in Section 3.1, we turn our attention to MR.
The MR metric reflects the model’s ability to predict the non-zero moments of the distribution and as shown in Figure 1c all models learn to do this relatively well.
This is also true for the generalization setting (Figure 1d) in which models are forced to make predictions over unseen moments .
To gain more insight into the distributions induced by the models, another way of breaking down the MSPC metric is to focus on how well a model ranks the non-zero moments of the gold distribution.
That is, in this case, we tell the model which moments are non-zero and let the model rank them.
Figure 1e shows MSPCP.
We can see that all models learn to rank the moments of the target class and [Figure] this also holds true for unseen moments (Figure 1f).
In this evaluation scenario, the WFA seems to be better than the other two models.
This is most likely due to the difference between the loss function minimized by the WFA and the loss functions minimized by the LSTM and BERT models.
The WFA classifier is defined as an ensemble of class specific models, and therefore it is explicitly modeling the class distributions.
Finally we also explored correlations between different models by computing pairwise Spearman correlation between the model’s predictions.
We found out that the correlation is high for shorter moments and that it decreases significantly with longer ones.
Regarding pairwise associations, LSTM and BERT show a higher level of agreement, while the WFA is more correlated to BERT overall.
Conclusion
In this paper we presented a novel evaluation framework for comparing sequence classification models and for testing the extent to which they can implicitly learn the target distribution.
Our results comparing three different deep-learning models suggest that the models have the ability to learn the underlying distribution.
They also suggest that despite significant differences in the architectures and even in the loss functions (i.e. for the WFA), their performance is comparable.
The goal of this paper was to develop a simple way to gain insights into what is actually been learned by a given sequence classifier.
But the evaluation approach itself also suggests ways in which we could exploit unlabeled data to obtain better calibrated sequence classifiers, future work should explore this possibility.
Acknowledgments and Disclosure of Funding
The authors gratefully acknowledge the computer resources at ARTEMISA, funded by the European
Union ERDF and Comunitat Valenciana as well as the technical support provided by the Instituto de Física Corpuscular, IFIC (CSIC-UV).
This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement
No.853459 ).
A Optimization Details
For each of our training sets T m = { ( x (1) , y (1) ) , . . .
, ( x ( m ) , y ( m ) ) } we train a new model and validate the parameters independently.
We use a 2 layer LSTM and we validate all parameters such as the dimension of the hidden state, of the embeddings and the learning rate with a grid search evaluated on the validation data.
We proceed analogously for the BERT parameters, note that we use the bert-base model.
We use Adam [Kingma and Ba, 2014] as optimizer employing the standard cross entropy loss function.
For both LSTM and BERT we train 5 models with different starting random seeds for each training set and we report the average of the results.
[Table]
For the WFA model the only parameter that we need to validate is the number of states k .
Besides, as there are no randomly initialized parameters and the training is not stochastic, we don’t need to perform multiple runs with multiple seeds.
The WFA is trained using the spectral learning method.
For a detailed and comprehensive explanation of the training method the reader is referred to Balle et al.
Furthermore, we employ the scalability techniques described in Quattoni et al.
LSTM and BERT models were trained on a single GPU while the WFA, for which the bottleneck is computing a singular value decomposition of a large sparse matrix, was trained on a CPU.
To give an idea of the relative size of each model, we computed the minimum number of parameters for which the models were able to achieve perfect classification on our biggest training set.
We report the numbers in Table 1.
B Data
Here we presents some statistics about the data that we have generated.
There are a total of 23 amino acid symbols.
[Table]
- |
[Table]
C Models correlation
Here we show a figure displaying the pairwise Spearman correlation between the model’s predictions.
N i stands for the different ngram lengths.
D Result Tables
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]