Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers
Abstract
While neural network binary classifiers are often evaluated on metrics such as
Accuracy and F 1 -Score, they are commonly trained with a cross-entropy objective.
How can this training-evaluation gap be addressed?
While specific techniques have been adopted to optimize certain confusion matrix based metrics, it is challenging or impossible in some cases to generalize the techniques to other metrics.
Adversarial learning approaches have also been proposed to optimize networks via confusion matrix based metrics, but they tend to be much slower than common training methods.
In this work, we propose a unifying approach to training neural network binary classifiers that combines a differentiable approximation of the Heaviside function with a probabilistic view of the typical confusion matrix values using soft sets.
Our theoretical analysis shows the benefit of using our method to optimize for a given evaluation metric, such as F 1 -Score, with soft sets. Also, our extensive experiments show the effectiveness of our approach in several domains.
Introduction
Neural network binary classifiers output a probability p ∈ [0 , 1] which is often used at training time to optimize model parameters using the binary cross-entropy (BCE) loss.
The network’s output p can also be translated to a binary value { 0 , 1 } indicating set membership to the negative or positive class.
To determine set membership, the Heaviside step function H is commonly used with a threshold τ where p ≥ τ are considered positive classification outcomes.
This notion of set membership is often used in evaluation metrics for binary classifiers.
It is a common assumption that optimizing a network via the desired evaluation metric is preferable to optimizing a surrogate objective [12, 18, 29, 33].
Unfortunately, optimizing a network using the desired evaluation metric, such as F 1 -Score, is typically not feasible.
The Heaviside step function used to compute confusion matrix set membership in terms of true positives, false negatives, false positives, and true negatives, has a gradient with properties not conducive to optimization via gradient descent.
The Heaviside function’s gradient is not defined at the threshold τ and is zero everywhere else.
Empirical results (e.g., [12]) show that the training-evaluation gap hinders evaluation performance. The gap means that in many applications, the metric differs significantly from the surrogate loss [33] and there may not be a strong correlation between minimizing a surrogate loss and improving an evaluation metric [29]. Also, it could mean that classifiers are solving “the wrong problem” when optimizing a surrogate loss, leading to sub-optimal evaluation performance [18].
To address the gap between training and evaluating neural network binary classifiers, we propose a method to make confusion-matrix based evaluation metrics usable for backpropagation.
Specifically, we propose the use of a differentiable approximation of the Heaviside step function along with confusion matrix values computed using the notion of soft set membership.
A desired evaluation metric can then be made differentiable by calculating it over the soft-set confusion matrix, rather than using the traditional confusion matrix values.
Our main contributions are: 1) a novel method for training neural network binary classifiers that allows for the optimization of confusion-matrix based evaluation metrics with soft sets (Sec. 3); 2) a theoretical analysis of our method (Sec. 4); and 3) the application of our approach to various domains with varying levels of class imbalance, showing its flexibility and superior performance compared to several baseline methods (Sec. 5).
We provide an open-source implementation of our method for reproducibility.
Preliminaries
In binary classification via neural networks, a step function is required to transform the network’s output to a binary value.
A common choice is the Heaviside step function with a threshold value τ :
H ( p, τ ) =
(cid:40) 1 p ≥ τ 0 p < τ (1)
Confusion matrix set membership is then computed for a prediction p and ground truth label y via: tp ( p,y,τ ) =
(cid:40) H ( p,τ ) y
= otherwise fn ( p,y,τ )
=
(cid:40) 1 − H ( p,τ ) y
= otherwise fp ( p,y,τ ) =
(cid:40) H ( p,τ ) y
= otherwise tn ( p,y,τ ) =
(cid:40) 1 − H ( p,τ ) y
= otherwise (2)
Consider a set of predictions p ∈ [0 , 1] , ground truth labels y ∈ { 0 , 1 } and threshold value τ ∈ (0 , 1) , e.g., { ( p 1 , y 1 , τ ) , ( p 2 , y 2 , τ ) , ..., ( p n , y n , τ ) } .
For n samples, the cardinality of each confusion matrix set is then computed as:
[Equation]
Common classification metrics are based on these cardinalities.
For example, Precision = | TP | / ( | TP | + | FP | ) is the proportion of positive predictions that are true positive results.
Recall = | TP | / ( | TP | + | FN | ) indicates the proportion of positive examples that are correctly identified.
These two metrics represent a trade-off between classifier objectives; it is generally undesirable to optimize or evaluate for one while ignoring the other [17].
This makes summary evaluation metrics that balance between these trade-offs popular.
Commonly used metrics include:
[Equation]
Accuracy is the rate of correct predictions to all predictions.
F 1 -Score is a specific instance of
F β -Score, which is the weighted harmonic mean of precision and recall: F β -Score =
(1 + β 2 ) · ( precision · recall ) / ( β 2 · precision + recall ) .
The value β indicates that recall is considered β times more important than precision.
Some metrics like F β -Score are usually computed at a specific threshold τ whereas others are computed over a range of τ values.
For example, the area under the receiver operating characteristic curve (AUROC) is a commonly used ranking performance measure defined in terms of the true positive rate (TPR) and false positive rate (FPR), each a function of τ : (cid:82)
−∞ ∞
TPR ( τ ) FPR ′ ( τ ) dτ .
While metrics that rely on confusion matrix set values are commonly used for evaluation, as shown in
Fig.
1 (right), it is difficult to use them as a loss during training.
These metrics rely on the Heaviside [Figure] [Figure] step function, the derivative of which is undefined at the threshold τ and zero everywhere else.
This means that these metrics do not have a derivative useful to backpropagate errors through the network.
When a binary classification neural network is trained on a loss function that is different from the evaluation metric, such as BCE, the network parameters are unlikely to be optimal for the desired evaluation metric.
As shown in our theoretical analysis and experimental results, when the goal is to balance various confusion matrix set values, e.g., with F 1 -Score, performance is improved by using our proposed method to optimize F 1 -Score.
Method
We propose a method (Fig. 1) that aims to better unify the training and evaluation steps of binary neural network classifiers whose performance is measured with metrics based on the confusion matrix.
Our method has two main steps.
First, the Heaviside step function, H , is approximated with a function H useful for optimization via gradient descent.
Then, H is used to compute a soft version of set membership in the confusion matrix.
With the soft-set confusion matrix values, we can then compute desired confusion-matrix based metrics, such as F 1 -score, using their standard formula.
This appraoch makes the metrics end-to-end differentiable so that they can be used as a training loss.
3.1 Heaviside Approximation
A useful Heaviside approximation, H , has a non-zero gradient: H ′ ( p, τ ) ̸ = 0 , ∀ τ .
Also, it ensures that for a single example, the classifier predicts positive and negative probabilities which sum to 1.
Like H , the approximation should meet the properties of a cumulative distribution function (CDF) which ensures it is right-continuous, non-decreasing, with outputs in [0 , 1] following:
[Equation]
Sigmoid approximation: One approximation method for H , proposed by [23], is the sigmoid function s 0 ( k ; p ) =
(1 + e − kp ) − 1 .
We reparameterize s 0 to account for τ , so that it can be used for varying thresholds: H s ( k, p, τ ) =
(1 + e − k ( p − τ ) ) − 1 .
A challenge with the sigmoid approximation is that as k increases, the sigmoid function better approximates the Heaviside function, but the derivative is close to zero over a larger range of valid inputs.
Another challenge is that when τ is close to 0 or 1, H s does not approach H as the input approaches 0 or 1.
These limitations as well as the hyperparameters k and τ are further discussed in the Supplementary Material (Sec. 2).
Linear approximation:
An alternative approximation of H is a five-point linearly interpolated function (Fig. 1, left).
This approximation H l , is defined over [0 , 1] and is parameterized by a given threshold τ and a slope parameter δ , which define three linear segments with slopes m 1 , m 2 , and m 3 .
The slope of each line segment is:
[Equation]
with τ m = min { τ, 1 − τ } in order to ensure a gradient suitable for backpropagation.
The linear Heaviside approximation is thus given by:
[Equation]
Considering the threshold τ in the formulation of H l ensures H l ( p = τ, τ ) = 0 .
5 while maintaining the limits in Eq.
See the Supplementary Material, Sec. 2.2, for the derivation of Eq.
Recently, a similar linear function was implemented as an activation function to enforce meaningful logical outputs in logical neural networks [30].
However, rather than using the linear function for activation within the larger model architecture, we use it in the training objective of a classifier.
3.2 Soft Sets
We use soft sets, a generalization of fuzzy sets [25], to compose differentiable versions of confusionmatrix based evaluation metrics useful for backpropagation.
A soft set µ in U , the initial universal set, is defined by the membership function µ : U → [0 , 1] .
For x ∈ U , the set membership function µ ( x ) specifies a degree of belonging for x to set u .
We use the notion of soft sets to compute confusion-matrix based metrics with a Heaviside approximation in place of the typically used strict sets.
Soft set membership corresponds to the degree to which a sample tuple ( p, y, τ ) belongs to a confusion matrix set.
We define the soft confusion matrix set membership functions for prediction and ground truth examples ( p, y ) relative to τ as: tp s ( p,y,τ ) =
(cid:40) H ( p,τ ) y
= otherwise fn s ( p,y,τ ) =
(cid:40) 1 − H ( p,τ ) y
= otherwise fp s ( p,y,τ ) =
(cid:40) H ( p,τ ) y
= otherwise
tn s ( p,y,τ ) =
(cid:40) 1 − H ( p,τ ) y
= otherwise (6)
Whereas the values of the confusion matrix ( tp , fn , fp , tn ) are the sum of zeros and ones, and the values of the soft confusion matrix ( tp s , fn s , fp s , tn s ) are the sum of continuous values in [0 , 1] , rather than integers.
After computing per-sample soft set values, specific metrics can be approximated by summing over the relevant elements of the confusion matrix.
For instance, we approximate precision as | TP s | / ( | TP s | + | FP s | ) , where | TP s | = (cid:80) ni
=1 tp s ( p i , y i , τ ) and | FP s | = (cid:80) ni
=1 fp s ( p i , y i , τ ) for n samples.
In practice, this summation occurs at training time over mini-batches while optimizing via gradient descent.
Because gradient descent and its variants expect a small but representative sample of the broader data [5], the proposed method also expects a representative sample.
Decomposability: Metrics based on confusion matrix set values are considered non-decomposable.
Non-decomposable metrics cannot be calculated per datapoint and are not additive across subsets of data [19].
We acknowledge that our optimization method of mini-batch stochastic gradient descent (SGD) does not provide an unbiased estimator on non-decomposable metrics.
However, this is common in neural network training [16].
In practice, we find that large enough mini-batches provide a representative sample for confusion matrix based metrics, allowing the use of mini-batch SGD in our approach.
Moreover, our method does not limit maximum batch size for training, unlike the adversarial approach to optimize F 1 -Score proposed by [14].
Metrics and losses: Any metric composed of confusion matrix set values ( TP , FP , FN , TN ) can be approximated using our proposed method and used as a training loss.
In our experiments, we train and evaluate on Accuracy, F 1 -Score, AUROC, and F β -Score [31].
We chose these metrics to show the flexibility of our approach.
Accuracy, F 1 -Score and F β -Score may be selected based on class imbalance and resulting tradeoffs.
AUROC illustrates that our method can be used when computation over a range of thresholds, τ , is required.
Theoretical Grounding
In this section, we first show the Lipschitz continuity of a variety of soft-set based metrics under the proposed Heaviside function approximations.
This indicates that when such metrics are used as the loss in a neural network, the difference between successive losses is bounded across iterations of stochastic gradient descent.
We then show, under certain assumptions, that metrics computed over the soft-set confusion matrix values are asymptotically similar to the true metric.
4.1 Lipschitz Continuity of Metrics Based on Soft-Set Confusion Matrix Values
A function f is considered Lipschitz continuous if there exists some constant K such that for all x 1 , x 2 in the domain, | f ( x 1 ) − f ( x 2 ) | ≤ K | x 1 − x 2 | .
Lipschitz continuous functions are always themselves continuous.
Theorem 4.1.
The linear Heaviside approximation H l is Lipschitz continuous with Lipschitz constant
M = max { m 1 , m 2 , m 3 } .
H l is continuous because each piecewise linear component is continuous and H l is defined to be continuous at the two points p =
τ ± τ m 2 .
Also, by construction, the slope of any secant line is positive and bounded by M = max { m 1 , m 2 , m 3 } .
Thus, H l is Lipschitz continuous with Lipschitz constant M .
Please see the full proof in Sec.
1.1 of the Supplementary Material.
Note that the H s is also Lipschitz continuous [32].
Theorem 4.2.
Every entry of the soft-sets confusion matrix based on the Heaviside approximations is Lipschitz continuous in the output of a neural network.
The proof of Theorem 4.2 is in the Supplementary Material (Sec. 1.1).
The value of a Lipschitz continuous loss function composed of soft-set confusion matrix values is Lipschitz continuous.
This is because compositions of Lipschitz continuous functions are also
Lipschitz continuous.
Following [9], we note that the confusion-matrix based metrics Accuracy,
Balanced Accuracy, F β , Jaccard, and G-Mean are all Lipschitz continuous.
These metrics are also
Lipschitz continuous under our proposed method computed via soft sets.
Lipschitz continuity of the loss function in a neural network optimized via stochastic gradient descent indicates convergence without extreme variations in losses throughout training.
Let K be the Lipschitz constant for the objective function ℓ ( w ) on network weights w .
Because of Lipschitz continuity, when updating w i +
→ w i − α i ℓ ′ ( w i ) using stochastic gradient descent with a learning rate α i , a small local change in the weights | w i {++
+} − w i |
= α i | ℓ ′ ( w i ) | corresponds to a small local change in the value of the objective function of | ℓ ( w i +1 ) − ℓ ( w i ) | ≤ Kα i | ℓ ′ ( w i ) | .
4.2 Approximation of Confusion-Matrix Based Metrics with Soft Sets
We provide a statistical analysis showing that in the limit, as the number of examples goes to infinity,
F 1 -Score calculated with soft sets approximates the expected true F 1 -score under a set of assumptions.
Similar proofs for Accuracy and AUROC are provided in the Supplementary Material (Sec. 1.2).
Consider a dataset of size n with { x 1 , ..., x n } examples and { y 1 , ..., y n } labels.
Suppose a network outputs a probability p i that the label y i
= 1 .
In this section, since the specific outputs p i are unknown and may change across iterations, we assume p i is a random variable.
When calculating F 1 -Score, p i is passed through the Heaviside function H , which generates an output ˆ y
Hi =
H ( p i , τ ) , where ˆ y Hi ∈ { 0 , 1 } .
The F 1 -Score can be expressed as:
[Equation]
To calculate F 1 -Score with soft sets ( F s 1 ), p i is passed through the Heaviside approximation H , which generates an output ˆ y H
i
= H ( p i , τ ) , where ˆ y H i ∈ [0 , 1] .
F 1 -Score with soft-sets can be expressed as:
[Equation]
Consider a network trained on a dataset with rn positive and (1 − r ) n negative elements, where r ∈ [0 , 1] is some constant.
Suppose this classifier correctly classifies any positive example as a true positive with probability u and any negative example as a false positive with probability v .
Also, assume that all classifications are independent.
Because F 1 -Score is calculated with discrete ˆ y Hi , we assume that the classifier will classify examples as a random variable ˆ y
Hi ∼ Bernoulli
( uy i + v (1 − y i )) .
Thus, ˆ y Hi ∼ Bernoulli( u ) if y i = 1 , and ˆ y
Hi ∼ Bernoulli
( v ) if y i = 0 .
Since F s 1 can take on continuous values in [0 , 1] , we consider that ˆ y H
i is a random variable drawn from a Beta distribution, which has support [0 , 1] .
In particular, assume ˆ y H
i ∼ Beta( α u y
i + α v (1 − y i ) , β u y i
+ β v (1 − y i )) .
Hence, ˆ y H i ∼ Beta( α u , β u )
if y i
= 1 , and ˆ y H
i ∼ Beta( α v , β v ) if y i = 0 .
Let
[Equation]
Under the above assumptions, both F 1 and F s 1 have the same average classification correctness: for any given i , E (cid:2)
ˆ y Hi (cid:3) =
E (cid:2)
ˆ y H
i (cid:3) .
Also, there exists some α u , β u , α v , β v such that the distributions of ˆ y
Hi =
H ( p i , τ ) and ˆ y H
i
= H ( p i , τ ) can both hold simultaneously under the same network for all i .
If we let U ∼ Binomial( nr, u ) and V ∼ Binomial( n (1 − r ) , v ) be the independent random variables denoting the number of true positives and false positives in a sequence of n independent predictions, then F 1 from Eq.
(7) becomes:
[Equation]
By the Strong Law of Large Numbers, 1 nr U a.s. → u and 1 n (1 − r ) V a.s.
→ v both converge with probabilityas n → ∞ .
Hence, Un a.s. → ru and Vn a.s.
→ (1 − r ) v .
We therefore have, from the Continuous Mapping
Theorem, that as n → ∞ :
[Equation]
For F s 1 , let U s =
(cid:80) y
i
=
1 ˆ y H
i and V s = (cid:80)
y i =
ˆ y H
i be the independent random variables denoting the total amount of true positives and false positives in the soft set case.
Then, F s 1 from Eq.
(8) becomes:
[Equation]
Since U s is the sum of nr i.i.d. random variables distributed as Beta( α u , β u ) , which has mean α
[Equation]
+ β
[Equation]
, by the Strong Law of Large Numbers, 1 nr U s a.s.
→ α u α u + β u = u .
Similarly, 1 n (1 − r ) V s a.s. →
[Equation]
α v + β v = v also converges with probability 1 as n → ∞ .
Hence, U s n a.s. → ru and V s n a.s.
→ (1 − r ) v .
We therefore have, from the Continuous Mapping Theorem, that as n → ∞ ,
[Equation]
Thus, F 1 and F s 1 both converge almost surely to the same value as n → ∞ .
Since 0 ≤ [Equation] 1 is bounded, E [ F 1 ] , E [ F s 1 ] → 2 ru r + ru + v − rv by the Bounded Convergence Theorem.
This means that the F s 1 value is an asymptotically unbiased estimator for the expected true F 1 -Score, and we expect average F 1 -Score values to converge to F s 1 as n → ∞ , under our setup.
However, the F 1 -Score computed from soft sets is not an unbiased estimator for the expected true F 1 -Score for finite n .
The proof of almost sure convergence generalizes for any metric that is a continuous function in the ratio of each entry of the confusion matrix to n , as shown in the Supplementary Material (Sec. 1.2).
This suggests that optimizing over the desired evaluation metric using our proposed method (e.g.,
F 1 -Score computed with soft sets) is justified, as the loss should follow our true final loss closely for a large enough n .
However, note that some metrics may be poor indicators of true classifier performance, such as Accuracy when data is imbalanced [17].
In such cases, it may not be desirable to optimize for such metrics using our method, as further discussed in the next section.
Experiments
This section presents three experiments to (1) evaluate the performance of our approach against several baselines on tabular data, (2) evaluate our approach on higher-dimensional image data, and (3) evaluate the ability of our method to balance precision and recall during training.
Datasets: Experiments were conducted on five publicly available datasets in a variety of domains and were chosen for their varying levels of class imbalance as explained later.
All datasets were minimally pre-processed and split into separate train, test, and validation sets.
See Sec.
4.1 of the
Supplementary Material for details.
Architecture and training: Our experiments aim to fairly evaluate our method using different confusion-matrix based objective functions.
Therefore, the same network architecture and training scheme was used unless otherwise noted.
Performance was evaluated over 10 repeated trials to control for the effects of random weight initialization.
We report the mean of results evaluated over threshold values T = { 0 . 1 , 0 .
2 , ..., 0 . 9 } for metrics that require a threshold choice at evaluation.
The details of the network architecture and training scheme are in Sec.
4.2 of the Supplementary Material.
Baselines: We trained networks using the typical BCE loss as well as two existing approaches for optimizing specific confusion matrix metrics: an adversarial approach for F 1 -score [14], and using an approximation of the Wilcoxon-Mann-Whitney (WMW) statistic for AUROC [39].
Also, the Supplementary Material provides comparisons with other, less related methods for binary classification.
5.1 Experiments on Tabular Data
We evaluate our proposed approach using four tabular datasets with different levels of class imbalance.
The CocktailParty dataset [40] has a 30.29% positive class balance making it the most class-balanced dataset of those considered in this experiment.
Salary classification data in the Adult dataset, from the UCI Machine Learning Repository [11], has a 23.93% positive class balance.
Classifications of microcalcifications in the Mammography dataset [38] is heavily skewed with a 2.32% positive class balance.
Lastly, the Kaggle Credit Card Fraud Detection dataset [36] has the most extreme class balance with only a 0.17% positive class balance.
We compare baseline methods against the performance of neural networks trained using our method to optimize F 1 -Score, Accuracy, and AUROC.
We instantiated our method using the Heaviside approximations and soft sets.
Results using the linear approximation are presented in Table 1.
See the
Supplementary Material, Sec. 5, for results using the sigmoid approximation, which were comparable.
F 1 computed with soft sets.
Our method of optimizing F 1 over the soft-set confusion matrix outperforms baselines when evaluated on F 1 -Score for all datasets.
In Table 1, line (1) has a higher
F 1 -Score than lines (2) and (6).
Additionally, even when evaluated on the other metrics (Accuracy and AUROC), networks trained using our method on F 1 perform similarly or better than BCE.
Accuracy computed with soft sets.
Our method of optimizing Accuracy over the soft-set confusion matrix has comparable performance to the BCE baseline when evaluated on Accuracy for all datasets.
In Table 1, line (3) has comparable Accuracy to line (6).
Note that Accuracy can reward prediction of only the dominant-class in imbalanced datasets [17], potentially resulting in no positive predictions and an F 1 -Score of 0, as in line (3) of Table 1 for the Mammography dataset. Our theoretical analysis (Sec. 4.2) indicates that our proposed method of optimizing confusion-matrix based metrics using soft-set F 1 or soft-set Accuracy should converge to the evaluation metric ( F 1 or Accuracy respectively) in the limit. The alternative of using BCE loss performs comparably to our proposed method in some cases; however, BCE = − 1 n (cid:80) ni
{+=
 ( y i log p i + (1 − y i ) log(1 − p i )) and Accuracy = ( | TP | + | TN | ) /n are different expressions that don’t generally converge to each other as n → ∞ .+} AUROC computed with soft sets.
Neither optimizing AUROC using our method nor optimizing
AUROC via the WMW statistic [39] consistently outperforms traditional BCE.
In Table 1, lines (4) and (5) have lower F 1 -Score, Accuracy, and AUROC than line (6).
This may be due in part to the
AUROC metric’s challenge with scale assumptions [15].
Moreover, AUROC is not linear-fractional [21], unlike F β -Score and Accuracy.
Therefore, AUROC does not necessarily have an unbiased estimator of gradient direction [1], which may also contribute to lackluster performance.
[Table]
[Table] Note that in comparison to [14], the adversarial approach F 1 † did not perform as well in our experiments.
The difference could be due to different datasets, and because we used a PyTorch implementation provided by the author while [14] utilized Julia and Flux ML.
Training networks for the tabular datasets on F 1 -Score using our method on a CPU took a median time of 2 . 3 ± 0 .minutes whereas the adversarial approach on CPU took a median time of 70 ± 118 minutes.
5.2 Experiments on Image Data
We conducted experiments similar to the those in Sec.
5.1 with higher dimensional data using two different binary image datasets created from the CIFAR-10 dataset [22].
The CIFAR-10Transportation and CIFAR-10-Frog datasets had a 40% and 10% positive class balance, respectively.
We focused on Accuracy and F 1 -Score, given the limitations with AUROC found in earlier experiments.
We also excluded the adversarial approach to optimize F 1 -Score [14] due to long runtime on tabular data.
Otherwise, we use the same set up from Sec. 5.1.
Overall, our findings with image data in Table 2 are consistent with tabular results from Sec. 5.1.
Our method of optimizing F 1 -Score over the soft-set confusion matrix performed better than traditional
BCE for both datasets when evaluated on both F 1 -Score and Accuracy.
In Table 2, line (1) has higher scores than line (3).
Results from our method of optimizing Accuracy over the soft-set confusion matrix (line (2) in Table 2) are better than the results for BCE (line (3) in Table 2). 5.3 Experiments on Graph Data We applied our method to a more structured representation of the examples in the CocktailParty dataset [40]. Following Thompson et al.
[35], we constructed a fully-connected graph in which each following Thompson et al.
F 1 and proposed [Table] [Table] person corresponded to a node in the graph and edges held distances between people.
We then used a message-passing Graph Neural Network (GNN) [2] to predict pairwise affinities corresponding to whether two individuals were part of the same conversational group.
This task was similar to the experiment on tabular data (Sec. 5.1) using the CocktailParty dataset
[Table]
but instead of predicting interactions among pairs of potential interactants independently, we made predictions for all the pairs seen in a frame of the dataset simultaneously.
Using the hyperparameters proposed by Thompson et al.
[35], we obtained results (Table 3) in line with those for the tabular data (Sec. 5.1).
5.4 Balancing Between Precision and Recall
Our method also allows training-time optimization that balances between precision and recall using
F β -Score.
This approach to training is particularly useful in real-world scenarios where there is a high cost associated with missed detections.
This type of metric is difficult to optimize effectively for using a typical BCE loss because BCE is not aware of any preference towards precision or recall.
Results in Table 4 show that using the proposed method to optimize F β -Score is an effective way of maintaining maximum classifier performance while balancing between precision and recall at a ratio appropriate for a given task.
Increasing values of β correspond to an increased preference toward recall with small loss of total performance measured by F 1 -Score.
Similar results on optimizing for
F β -Score with additional datasets are provided in Sec.
6.2 of the Supplementary Material.
Related work
Our work is inspired by research on the direct optimization of evaluation metrics for binary classification.
This includes plug-in methods that empirically estimate a threshold for a classifier on a metric.
For example, [26] demonstrated the applicability of plug-in classifiers to optimize F 1 -Score with linear models.
For metrics based on linear combinations of confusion matrix set cardinalities, [21] identified an optimal plug-in classifier with a metric-dependent threshold.
Also, [20] explored the
Precision@K metric for linear models in the context of ranking.
Our approach is not a competitor to plug-in methods, but rather an approach to train a neural network classifier on a differentiable approximation of a metric based on the confusion matrix.
As such, it could be used in conjunction with a plug-in method, if desired.
Works such as [7, 19] optimized specific metrics like Precision@K and F 1 -Score in online learning, which is characterized by the sequential availability of data.
Our work does not address online learning, but batch learning methods.
Additionally, other work has focused on optimizing AUROC [18, 39], F-score [10, 41], and AUPRC in the context of ranking [12].
Rather than focusing on a single specific metric, we provide a flexible method for optimizing a neural network using approximations of varied metrics based on the confusion matrix.
For example, our experiments in Sec.
5.4 show how tradeoffs between precision and recall can be made by adjusting the β parameter of the F β -score during training with our method.
In the field of computer vision, differentiable surrogate losses have been proposed for the F 1 -Score [6, 8], Jaccard Index [3, 28, 29], and Dice score [4, 24, 27, 34].
However, since these methods are applied to difficult problems in computer vision, they incorporate a particular surrogate, such as a surrogate for the F 1 -Score, into a larger composition of losses.
Mean average precision (mAP) [13], for example, is composed of a number of intersection-over-union (IoU) values at different thresholds for multiple classes.
We leave the study of metrics which require complex aggregation to future work.
In our experiments, we focus on evaluating the impact of individual losses approximating a desired evaluation metric.
We also evaluate the theoretical merits of optimizing a differentiable surrogate for confusion-matrix based metrics using an approximation of the Heaviside step function and soft-set confusion matrix values.
Recently, adversarial approaches have emerged as another related area of research.
Wang et al. [37] used a structured support vector machine and reported performance on Precision@K as well as F 1 -Score.
Fathony and Kolter [14], which we compare against, improved performance via a marginalization technique that ensured polynomial time convergence.
The authors evaluated their adversarial approach on Accuracy and F 1 -Score, among other metrics, while reporting performance relative to BCE [14].
Downsides of the latter approach are that it is limited to a small batch size, on the order of 25 samples, and has cubic runtime complexity.
Our approach does not limit batch sizes and has a worst-case runtime complexity equivalent to the runtime complexity of the confusion-matrix based metric being used as a loss.
Broader impact and ethics
Our work was motivated by the potentially wide ranging impact that more flexible and robust binary classifiers could have across application domains such as social group dynamics, sociology, and economics, which we explored in our experiments.
Improving the tools used in these areas of research has the potential to positively impact human quality of life, but are still susceptible to data bias.
These tools should be used with care when building safe and responsible artificial intelligence systems.
Conclusion
We proposed a novel method to optimize for confusion-matrix based metrics by using a Heaviside function approximation and soft-set membership.
Our method addresses the common trainingevaluation gap when working with binary neural network classifiers: these networks are typically trained with BCE loss, but evaluated using a confusion-matrix based metric, such as F 1 -Score.
Our theoretical analyses showed that soft-set confusion matrix based metrics, such as F s 1 , are Lipschitz continuous and are likely to converge in expectation to the true metric’s expectation.
Additionally, our experiments showed the feasibility of using our method to optimize confusion-matrix based metrics.
While many factors play into final classifier performance, such as balance of samples in the dataset, the desired evaluation metric, the approximation of H, and network hyperparameters, we found that in many cases, final classifier performance can be improved by bridging the training-evaluation gap using our method.
In fact, optimizing model parameters for F 1 with soft sets resulted in better evaluation results with F 1 -Score and better or comparable evaluation results with Accuracy and
AUROC.
Our approach also outperformed other methods that directly optimize for a specific metric.
For example, Fathony and Kolter’s [14] adversarial approach applied to F 1 -Score had much longer training time and resulted in worse performance than our approach.
In the future, we are interested in further investigation of other aspects of our approach such as analyzing statistical sample efficiency as well as the approximation error and the generalization error.
Further, we believe studying how our method can be applied to multi-class classification problems is an interesting area for future work.
Acknowledgements
The authors thank Andre Wibisono for helpful discussions and feedback.
This work was supported by the National Science Foundation (NSF), Grant No. (IIS-1924802).
The findings and conclusions in this article are those of the authors and do not necessarily reflect the views of the NSF.