Scalable design of Error-Correcting Output Codes using Discrete Optimization with Graph Coloring
Abstract
We study the problem of scalable design of Error-Correcting Output Codes (ECOC) for multi-class classification.
Prior works on ECOC-based classifiers are limited to codebooks with small number of rows (classes) or columns, and do not provide optimality guarantees for the codebook design problem.
We address these limitations by developing a codebook design approach based on a Mixed-Integer Quadratically
Constrained Program (MIQCP).
This discrete formulation is naturally suited for maximizing the error-correction capability of ECOC-based classifiers and incorporates various design criteria in a flexible manner.
Our solution approach is tractable in that it incrementally increases the codebook size by adding columns to maximize the gain in error-correcting capability.
In particular, we show that the maximal gain in error-correction can be upper bounded by solving a graph-coloring problem.
As a result, we can efficiently generate near-optimal codebooks for very large problem instances.
These codebooks provide competitive multi-class classification performance on small class datasets such as MNIST and CIFAR10.
Moreover, by leveraging transfer-learned binary classifiers, we achieve better classification performance over transfer-learned multi-class CNNs on large class datasets such as
CIFAR100, Caltech-101/256.
Our results highlight the advantages of simple and modular ECOC-based classifiers in improving classification accuracy without the risk of overfitting.
Introduction
Error-correcting codes have found many applications in machine learning.
Their use range from natural classification tasks [8, 24, 2, 17, 21] including multi-label classification [11] to robust classification [43, 46, 16] to federated learning [22] to zero-shot learning [40] to life-long learning [20].
The seminal paper of [8] proposed the use of error-correcting output codes (ECOCs) for classification tasks.
Their framework lead to a significant activity on the design of ECOCs with a variety of features; for e.g. the use of different class of encodings [7, 17, 39], hierarchical classifiers [14, 12], decoding schemes [2, 9] and accounting for the underlying data-distribution [49, 47, 30, 33].
However, a key challenge that remains to be addressed is a flexible and tractable approach for designing high-quality codebooks.
Our paper addresses this challenge.
We tackle some of the limitations of prior works on solving the discrete codebook design problem.
In particular, we depart from the classical approach that performs a continuous relaxation of the discrete codebook design problem and attempts to solve the resulting non-linear optimization problem [7, 49, 47, 33].
That approach has been shown to scale to large number of classes k , but under the limitation that the size of the codebook (i.e. number of columns L ) is small.
However, with small
L , codebooks do not fully exploit the error-correcting capability, which is the key motivation to use
ECOCs in the first place.
Besides, the continuous relaxation approach typically does not provide codebooks with strong optimality guarantees.
We formulate the codebook design problem as a mixed integer quadratically constrained program (MIQCP) and develop a greedy algorithm to efficiently solve this MIQCP.
Each iteration of our algorithm adds a new set of columns to the current codebook such that the gain in error-correcting capability is maximal.
Importantly, we exploit a graph-coloring based upper-bound to accelerate the iterations of the greedy algorithm.
Thus, our algorithm provides the flexibility to choose codebook size based on the desired level of final accuracy.
The key advantage of our approach is that it easily scales to large problem instances ( k = 500 ).
In comparison, the recent paper of [16] generates high-quality codebooks but only for k ≤ 50 .
Thus, our approach can be maximally leverage the benefit of ECOCs in aforementioned applications of error-correcting codes for realistic problem instances.
Our main contributions are as follows: (i)
In contrast to [8, 33, 49, 47, 16], we model the codebook design problem as an element-wise optimization problem (MIQCP formulation).
(ii) We develop a solution approach using a greedy algorithm to solve large instances of the MIQCP with low-optimality gaps.
(iii)
We make a theoretical connection to the vertex-coloring problem to upper bound the maximal gain in the error-correction which leads to faster iterations of the greedy algorithm.
(iv) Our codebooks achieve very high classification accuracy similar to multi-class CNNs on datasets with small number of classes.
(v) Finally, we show a novel benefit of ECOCs with transfer-learning on large class datasets.
Our codebooks outperform multi-class CNNs when trained with transfer-learning under both nominal and robust feature settings.
ECOC for Classification
The ECOC-based framework [8], encodes each class of a k -class classification problem with a unique codeword of length L .
This encoding results in a codebook (coding matrix)
M of size k × L .
For binary codes: M ∈ { +1 , − 1 } k × L .
The rows (resp.
columns) of M correspond to distinct classes (resp.
binary classifiers or hypotheses ).
Figure 1 shows two example codebooks for a classification problem with 5 classes.
For every column in M a binary classifier is trained over the training data, where all training data from classes with entry +1 (resp.
entry − 1 ) forms the positive class (resp.
the other class).
[Figure]
For a test example x , let f 1 ( x ) , . .
, f L ( x ) denote the predicted class of each of the learned binary hypotheses, where f j ( x ) ∈ { +1 , − 1 } ∀ j ∈ { 1 , . . .
, L } .
This encoding F ( x ) , where F ( x ) =
[ f 1 ( x ) , . .
, f L ( x )] , is then associated with a class (i.e., a row of coding matrix M ), typically using a decoding scheme based on a similarity measure such as Hamming distance.
A common approach is to compute the Hamming distances d H ( · , · ) between F ( x ) and each codeword M ( i, · ) ∀ i ∈ { 1 , . . .
, k } and predict the output class ˆ y as the class with the minimum distance:
[Equation] [Equation]
j
[Equation]
(cid:18)
[Equation]
M
[Equation]
(cid:19)
ˆ
y = argmin
i
d H ( M ( i, · ) , F ( x )) .
Importantly, the final classification performance of the above ECOC procedure depends on both the accuracy of the trained binary classifiers and the error-correcting capability of the codebook M .
In particular this capability increases with the Hamming distances between rows; i.e. a codebook with high pairwise Hamming distances across its rows has a higher error-correction capability.
Definition 1.
Hamming distance of a codebook (denoted as κ M ) is defined as the minimum hamming distance between any two distinct pair of codewords (or rows) in M .
[Equation]
Proposition 1 (Error-Correction Capability) .
A codebook M with hamming distance κ M , can always correct at-least ⌊ κ M − 1⌋ errors.
Proof of proposition 1 and all subsequent technical results are provided in supplementary information (SI).
Problem Formulation
For a k -class classification problem we want to find an optimal codebook of length L .
Thus, we want to find the value of k × L entries of a codebook, where each ( i, j ) entry (denoted as x ij ) can take values in { +1 , − 1 } such that the resulting codebook has high multi-class classification accuracy.
To achieve high classification accuracy, we take the following design criteria [8, 16] into consideration:
The error-correcting property increases with the Hamming distance between rows; therefore it is desirable to have a high Hamming distance between any pair of rows.
We ensure this by maximizing - the Hamming distance of the codebook (or the minimum of the row pairwise Hamming distances).
For a binary codebook of size k × L , an analytical upper bound on this objective is given by the well-known Plotkin’s bound [37]: (cid:106)
Lk 2( k − 1) (cid:107) .
The Hamming distance between any two pair of columns of the codebook must be at leastto avoid same columns and at most k − 1 to avoid complementary columns.
A good column separation is desirable in order to avoid correlations between the resulting hypotheses (or binary classifiers).
Due to these reasons, we constrain the Hamming distances between any distinct pair of columns between ρ 1 and ρ 2 , where ρ 1 ≥ 1 and ρ 2 ≤ k − 1 .
Finally we consider the balanced column criterion.
This criterion allows to control the amount of imbalance in the resulting binary classification problem,
i
e. the number of data-points in each of the two classes.
This automatically ensures that no column has all entries as +1 or − 1 .
Importantly, while balanced columns help in improving the error-correcting capability, imbalanced columns often result in easier binary classification problems, especially when working with linear classifiers, thus promoting higher multi-class accuracy [2, 9, 41].
In SI, we
further discuss the nuances of shaping this trade-off in design of the codebook.
For notational convenience we denote the set of row indices as N , the set of column indices T , set of all pair of rows (or corresponding classes)
P N ; and the set of all pair of columns P T .
Mathematically:
[Equation]
We can now formulate the element-wise codebook design problem based on the above design criteria as the following Integer program ( IP 1 ):
[Equation]
The objective of IP 1 is to maximize the minimum Hamming distances between different pairs of rows (1a).
Constraint (1b) computes the Hamming distance between rows; (1c) ensures that column separation lies in the desirable range [ ρ 1 , ρ 2 ] ; (1d) ensures that every column results in a valid balanced binary classification problem, where n i is the number of training samples in i -th class of the dataset, while γ controls the degree of allowed imbalance; and finally (1e) ensures that x ij takes integer values in { +1 , − 1 } .
The max-min objective (1a) can be simplified by introducing an auxiliary variable t , where t :=
min { d 1 , 2 H ( x ) , d 1 , 3 H ( x ) , . . .
, d k − 1 ,k H ( x ) } and adding constraints t ≤ d 1 , 2 H ( x ) , . . .
, t ≤ d k − 1 ,k H ( x ) .
Note that (1b) and (1c) contain bilinear terms and are non-convex in nature.
Thus IP 1 can be categorized as a non-convex Mixed Integer Quadratic Constrained Program (MIQCP).
IP 1 cannot be solved directly as (1b) and (1c) contains bilinear terms and therefore these constraints are linearized to get a linear-relaxation of IP 1 .
Each bi-linear term is replaced with an auxiliary variable z ijpq , where z ijpq := x ij x pq .
Additional linear constraints known as McCormick-inequalities [34] are then added to lower and upper bound z ijpq .
Further details about McCormick inequalities (including their derivation) are provided in supplementary information (SI).
For the bilinear constraint z ijpq = x ij x pq , these inequalities are given by (2a) and (2b).
[Equation]
Also, (2a) and (2b) respectively provide convex lower and upper envelopes to the feasible set admitted by the bilinear constraint [1].
Lemma 1.
McCormick relaxation of IP 1 , denoted as MC ( IP 1) , is tight: (cid:8)
[Equation]
Thanks to lemma 1, IP 1 can now be solved by solving the MILP (i.e. MC ( IP 1) ) obtained after replacing the bilinear terms in (1b)-(1c) with the corresponding McCormick inequalities.
Note that
IP 1 has L × |P
N | + k × |P
T | =
L × (cid:0) k 2 (cid:1) + k × (cid:0) L 2 (cid:1) number of bilinear terms.
If L ≈ k , IP 1 will have O ( k 3 ) number of bilinear terms.
However in practice, linear programming (LP)-relaxation of
MC ( IP 1) can be quite loose and hence even with the tight reformulation, IP 1 cannot be used to tractably solve the optimal codebook design problem (especially when L is large).
Therefore, instead of generating optimal codebooks by solving IP 1 directly in one-shot, we next develop a scalable greedy solution approach to generate good-quality (near optimal) codebooks for large k and L .
Proposed Solution Approach
In this section we develop an iterative greedy algorithm to generate near-optimal codebooks i.e. codebooks with low-optimality gap.
To develop our approach, we utilize the following monotonicity result of lemma 2.
Lemma 2.
Let M and ˜ M be two binary error-correcting code of size k × l and k × ( l +˜ l ) respectively, where ˜ l ≥ 0 , such that all columns in M are also in ˜ M , i.e. M ⊆ ˜ M .
Then, the Hamming distances (recall 1) of M and ˜ M satisfy: κ M ≤ κ ˜ M , implying that the error-correcting capability of ˜ M is at least as good as M .
This intuitive result serves as the starting point of our solution approach.
In particular, from Lemma 2, we know that the error correcting capability of a codebook M of size k × l can be improved further by increasing the number of columns, i.e. adding ˜ l columns.
Naturally, this improvement not only depends on ˜ l , but also on how the entries of the new columns are selected.
Our solution approach progressively adds new columns to an existing codebook by solving a smaller integer program, denoted IP 2 ; see figure 2.
This integer program maximizes the hamming distance of the resulting codebook (i.e. its error-correcting capability), while ensuring that desired column separation for all (i.e. new and old columns) is achieved and the resulting codebook still remains a valid binary error-correcting code with balanced columns.
[Equation]

Solve
IP
[Figure]
[Equation]
Before proceeding, we introduce new notation for mathematical convenience.
Analogous to T , (resp.
P T ) let ˜ T denote the set of new columns (resp.
P ˜ T denote the set of pair of new columns).
Furthermore, let the entries of the new columns be denoted as ˜ x ij .
Mathematically these are defined in (3), (4) and (5) respectively.
We are ready to present IP 2 which takes a valid codebook M of size k × l and number of columns ˜ l to be added as inputs.
[Equation]
The objective function (6a) maximizes the the error-correcting capability.
Constraint (6b) computes the Hamming distance between pairs of rows (or classes) in the resulting codebook; (6c) ensures desired column separation between any pair of new columns; (6d) ensured desired separation between new columns and existing columns in input M ; (6e) ensures that new columns are balanced and (6f) ensures that the new codebook is a binary codebook.
Notice the key difference between IP 2 and IP 1 : since ˜ l ≪ l , the constraints (6d), which account for the majority of columns separation conditions in the resulting codebook are linear in ˜ x as the entries of M are known.
Further, the total number of
These features of IP 2 makes it solvable even for large k by off-the-shelf IP solvers which typically employ branch-and-bound (B&B) procedure.
It turns out that we can pre-compute a good quality (potentially tight) upper-bound to IP 2 .
In section 4.1, we explore two different ways to generate an upper bound to IP 2 .
Our computational results show that a good upper bound can accelerate the termination of the B&B procedure for IP 2 .
Since our greedy algorithm 4.2 relies on solving IP 2 repeatedly to increase the error-correcting capability, the overall computational gain can be significant.
4.1 Upper Bound to IP
The integer constraint (6f), ˜ x ij ∈ { +1 , − 1 } in IP 2 can be equivalently re-written as ˜ x 2 ij = 1 and the resulting problem can be categorized as a continuous , non-convex quadratic program.
We can generate an upper bound by taking the Rank-1 semi-definite programming (SDP) relaxation of this quadratic program (for details see SI).
The tightness of this relaxation is same as taking the dual of aforementioned quadratic program (equivalently IP 2 ), which is also a SDP and is dual to the RankSDP relaxation.
However, solving the SDP-relaxation can be expensive for large k even with modern first-order SDP cone solvers (for eg: SCS [36]).
More importantly our computational experiments suggest that the SDP-relaxation does not provide a good quality upper-bound to IP 2 .
We now discuss a second procedure to generate an upper bound to IP 2 .
To begin with, note that the addition of ˜ l columns to an existing codebook M , as done by IP 2 , can be equivalently viewed as the appending codewords (denoted as c i ) of length ˜ l to each row i of the codebook M , see figure 3.
For binary codes there are 2 ˜ l possible distinct codewords of length ˜ l , let the set of these codewords be denoted as C (˜ l ) .
Essentially in solving IP 2 , we seek to choose codewords from C (˜ l ) , such that the codebook resulting from appending them to the existing rows of M maximally increases the error-correcting capability.
In particular, out of all (cid:0) k 2 (cid:1) row pairs, IP 2 tries to increase the hamming distance between row pairs which are at the minimum in the existing codebook, as these minimum hamming distance row pairs decide the error-correcting capability (recall proposition 1).
For a given codebook M , we define the set of minimum distance row-pairs (denoted E M ) as:
[Equation]
For example, the minimum hamming distance between any two distinct pair of rows in the codebook shown in Figure 3 is 4 , and there are multiple row pairs for which the Hamming distance is 4 .
In
Codebook M 
 c
[Equation]
c
[Equation]
c 3 .
c k   where c i ∈ C (˜ l ) .
For ˜ l = 2 ,
C (2) :
= (cid:110)
[+1,-1] , [+1,+1] , [-1,+1] , [-1,-1] (cid:111)
[Figure]
general, for a codebook M of size k × L , we can represent the set E M as an undirected graph (denoted G
M ) with k -vertices, where vertex i corresponds to row i in M ∀ i ∈ { 1 , . . .
, k } .
For every ( i, j ) in E M , an edge is present between vertices i and j in G M .
We refer to graph G M as the error-correcting graph of M .
Mathematically,
G M = ( V , E M ) ,
where V = { 1 , . . .
, k }
Figure 3 illustrates a codebook M and its corresponding error-correcting graph G M .
This viewpoint enables us to make two important observations:
Increasing the hamming distance of row-pairs in E M by at least 1 through addition of columns (or equivalently by appending codewords c i ) to M , effectively increases the minimum hamming distance of the entire codebook by at least 1 . 2.
Any pair of distinct codewords in the set C (˜ l ) differ by at least 1 and at most ˜ l , i.e. : 1 ≤ d H ( c i , c j ) ≤ ˜ l
∀ c i , c j ∈ C (˜ l ) , i ̸
= j .
These observations lead to the following claim in proposition 2:
Proposition 2.
For a codebook M , if there exists an assignment of codeword c i ∈ C (˜ l ) to each vertex of graph G M such that no two connected vertices receive the same codeword, then the hamming distance of the codebook κ M can be increased by at least 1 by adding codewords from C (˜ l ) to the rows of M .
In fact the condition specified by proposition 2, can be easily verified by solving an instance of the graph-coloring problem on G M ,
where each codeword c i ∈ C (˜ l ) can be viewed as a unique color.
Before proceeding further, we recall some useful definitions related to graph coloring.
A proper-vertex coloring is an assignment of colors to the vertices of a graph so that no two adjacent vertices have the same color.
Definition 2 (Chromatic number) .
The chromatic number of a graph G , denoted as ξ ( G ) is the minimum number of colors required for a proper vertex coloring of the graph G .
Theorem 1.
For any binary code ˜ M resulting from adding ˜ l columns to an existing binary code M , the following holds regarding the hamming distance (recall 1) κ
˜ M of the code ˜ M :
• If the chromatic number of the graph G M , ξ ( G M ) is greater than the size of the set of possible codewords C (˜ l ) , i.e. ξ ( G M ) >
|C (˜ l ) | , then κ ˜ M = κ M .
• In particular, for ˜ l ∈ { 1 , 2 } * :
[Equation]
κ
[Equation]
if
[Equation]
Corollary 2.
An upper bound to IP 2 is provided by theorem 1 or IP 2 is infeasible.
Since for a given M , theorem 1 provides a valid upper-bound to the hamming distance (or equivalently the error-correcting capability) of all codebooks resulting from adding ˜ l columns to M , therefore it
1: Generate first column C 1 randomly.
2: M ← C 1 and start counter: i =3: while i <
L do 4:
Compute ξ ( G M ) .
Compute upper bound to IP 2 using ξ ( G M ) and ˜ l .
Solve IP 2 with latest M , ˜ l and upper-bound.
Update M with the solution of IP8:
i = i + ˜ l 9: end while 10: return M
[Table]
automatically also provides an upper-bound to IP 2 .
Further, the color-assignment can be used to generate a (partial) feasible solution to IP 2 .
Technical details of feasible solution generation are provided in SI.
Note that the problem of computing the chromatic number of a graph (or analogously solving the vertex-coloring problem) has been extensively studied in the literature.
This includes heuristics such as DSATUR [4] and RLF [28], exact backtracking algorithms [26] and more recent integer programming based exact methods [18, 35, 32].
We solve our graph-coloring problem by formulating and solving it as an integer program (IP); please refer to SI for details.
In table 1, we report the average time needed to solve the vertex-coloring problem for various error-correcting graphs G M with different k .
In almost all cases we are able to solve the graph-coloring problem extremely fast within 1-2 seconds.
This is mainly due to the highly sparse structure of G M as large number of nodes tend to have low or even zero-degree resulting from high diversity in row pair-wise Hamming distances.
4.2 Greedy Algorithm
We are now ready to present our Greedy Approach (Algorithm 1) for generating a codebook of size k × L which is a near-optimal feasible solution to our original codebook design problem.
Initially, we randomly pick the first column of the codebook such that it is a valid balanced binary classifier.
Then, a second column is added by solving IP 2 with ˜ l = 1 , this provides us with a codebook of size k × 2 that satisfies all the design criteria.
With this as our current codebook with two columns, we then continue to append more columns by again solving IP 2 .
This procedure is continued until we obtain a codebook with L columns.
Practically, it is important to keep ˜ l reasonably small so that the solution to IP 2 in each iteration of
Algorithm 1 is obtained quickly.
In the first few iterations, since M is small, therefore the number of column-separation constraints between the existing and new columns (6d) is small.
Thus, for large k , i.e. k > 50 , for first ⌈ log 2 k ⌉ † iterations we solve IP 2 with ˜ l = 1 and use ˜ l ≥ 2 in the subsequent iterations.
4.3 Flexibility in choosing the size of the codebook L
For most ECOC design approaches in literature, one needs to pre-define L to solve the codebookdesign and then train each of the binary classifiers.
If the resulting multi-class performance is not satisfactory then the process is repeated: increase L , resolve the codebook design problem and re-train all the columns in the new codebook.
Naturally, this iterative design process is quite burdensome because of the computation effort spent on solving and training codebooks in each iteration.
In a sharp contrast, our design approach does not require pre-specifying L as columns can be greedily added to build codebooks.
In particular, we can start off with a relatively small L , train the resulting columns and evaluate the multi-class performance.
If required, we can then append more columns to the existing codebook by solving IP 2 , train the new columns and re-evaluate the multi-class performance.
This procedure can be repeated until minimal or no improvement is observed.
Importantly the computation effort spent in training previous columns is not wasted and one can stop anytime if the improvement in the multi-class classification performance is negligible.
This flexibility also makes our approach dataset dependent to a certain extent [33].
Finally, we note that authors of [38, 39] have discussed choosing L but their work
We next perform computational experiments to evaluate the performance of our greedy algorithm and subsequently evaluate the classification performance of the codebooks generated by the greedy algorithm.
Experiments
We use Gurobi-v9.1 as our integer programming (IP) solver and run all our experiments on a machine with Intel Core i7-6800K CPU, 32GB RAM and 1080Ti Nvidia GPU.
We compute optimality gaps of codebooks generated by different approaches using Plotkin’s
Bound (PB).
The gap is defined as: | PB − f best | /f best , where f best denotes the objective function value of the solution.
Our code is available at: https://github.com/SamarthGM/Scalable_ecoc.
[Table] [Table]
We perform our first set of IP experiments on small problem instances ( k < 50 ) to provide a comparison between solving IP 1 directly and our Greedy approach (Algorithm 1).
Details regarding values of ρ 1 , ρ 2 , γ and their appropriate ranges are provided in SI.
We benchmark against the approach of [16] and also against randomly generated Dense codes.
The procedure to generate Dense codes [2] is discussed in SI.
We summarize our results in Table 2.
Notice that for very small instances such as k = 12 and k = 16 , IP 1 is able to generate a good quality solution, however for slightly larger problem instances (say k = 20 ) the solution quality of IP 1 deteriorates rapidly (the gap is around 50% ); and finally for k ≥ 24 , IP 1 simply fails to even generate a feasible solution.
This highlights the intractability of IP 1 to generate good quality codebooks.
In contrast, our Greedy approach provides very high quality solutions.
Note that for all cases, our greedy approach either matches or outperforms the column-subset selection (CSS) approach of [16].
On very small instances i.e. k ≤ 24 , [16] provides competitive performance to our greedy approach.
The reason behind this is because [16] provides a tight IP formulation but at the cost of having exponential O (2 k ) number of binary variables.
Their performance deteriorates after k ≥ 24 as they have to resort to random sampling to mitigate the computational expense of O (2 k ) number of binary variables.
It is not surprising that due to this exponential complexity, [16] is not tractable beyond k > 50 , as discussed next.
Also, Dense codes consistently have very high gaps in comparison to our Greedy Approach.
We evaluate the performance of our Greedy approach (Algorithm 1) on large problem instances, i.e. for k ≥ 50 .
These results are reported in Table 3.
We observe that approach of [16] does not even provide a feasible solution for k > 50 .
In contrast, our greedy approach generates good quality solution for even k = 500 while maintaining low optimality gaps.
For all cases ( k > 50 ) our gaps are well below 15%.
Thus our approach is significantly more tractable than of [16].
Interestingly, for k ≥ 350 Dense codes also provide competitive performance.
To the best of our knowledge, we are the first to achieve such scalability with low optimality-gaps for the codebook design problem.
5.1 Classification Performance
We now evaluate and benchmark the classification performance of our codebooks generated using greedy approach on different datasets.
For each dataset, we generate codebooks of different sizes using the Greedy approach and compare it with other commonly used codebooks like 1-vs-All, 1-vs- [41], Hadamard and Dense codes [2].
For every codebook we train L binary classifiers corresponding to the columns of the codebook.
We also benchmark against a multi-class CNN.
This multi-class
CNN has the same architecture as the binary classifiers except for the size of the output layer, which is 2 for binary classifiers and k for multi-class CNN.
For all experiments, we report average of 5 runs.
Details regarding different datasets, model architectures, hyper-parameters and training are provided in SI.
MNIST & CIFAR10:
In our first set of classification experiments, we evaluate the performance on
MNIST and CIFAR10 datasets (both with k = 10 ).
Using Greedy approach we generate codebooks of size L = 5 and 10 .
The classification performance of different codebooks is reported in tables 4 and 5 respectively.
On MNIST, the improvement in accuracy with increase in L is small, since
MNIST is a relatively easy classification task.
However, on CIFAR10, for which the underlying classification task is much harder than MNIST, we obtain a 1 . 8% gain in accuracy as the size of the codebook increases from L = 5 to L = 10 .
Further on CIFAR10, our compact greedy codebooks easily outperforms standard 1-vs-1 codebook which uses L = 45 binary classifiers by around 4 .
5% and also outperforms 1-vs-All codebook by more than 1% .
For both datasets, Dense and Hadamard codebooks also provides high accuracy, particularly Hadamard codebook on MNIST dataset.
In comparison to multi-class CNN, our greedy codebooks provides very similar accuracy on both datasets, with slightly lower accuracy on MNIST but higher on CIFAR10.
[Table] [Table]
CIFAR100 & Caltech-101/256:
In the second set of experiments we evaluate the performance on large class datasets: CIFAR100, Caltech-101 and Caltech-256 with k = 100 , 101 & 257 respectively.
Recall that, for these large class datasets, the approach of [16] cannot even generate a feasible codebook.
Thanks to our greedy algorithm, we can easily generate and evaluate our codebooks on these datasets.
Note that the computational expense associated with the training of L binary classifiers for each codebook can be large; however this is a general limitation of ECOC-based classifiers.
Firstly, our work addresses this limitation by generating compact codebooks with low-optimality gaps.
Secondly, we leverage the power of transfer learning to significantly reduce the training time of binary classifiers.
More generally, another motivation to leverage transfer-learning is due to the fact that modern deep-learning models continue to get bigger in size by the day and thus transfer-learning has become even more effective in the face of limited compute and data availability.
Our next results indeed demonstrate the additional benefit which ECOCs provide with transfer-learning to further improve classification accuracy.
For transfer-learning we use models trained on ImageNet and replace the last fully-connected layer of 1000 classes with a fully connected DNN with 2 output classes for binary classifiers and k classes for multi-class classifier respectively.
We then freeze the weights of all but the last fully-connected DNN.
Interestingly, authors in [42] and [45] have recently shown that adversarially trained robust models provide better generalization in target domain over nominally trained models.
Therefore for a more comprehensive evaluation of our approach, we use features from two different ResNet models trained on ImageNet.
The first model has been pre-trained nominally on the ImageNet dataset while the second model has been trained adversarially with l 2 -norm perturbations.
More details on these individual models are provided in SI.
The final multi-class accuracies of all the codebooks and multi-class CNN for both nominal and robust features are provided in tables 6, 7 and 8 respectively
[Table]
[Table] [Table]
We observe that on all the three (large class) datasets, our greedy codebooks consistently outperform multi-class CNNs.
For all cases, robust features indeed provide better accuracy over nominal features thus validating the hypotheses of [42, 45] in the context of ECOC-based classifiers.
Importantly, our greedy codebooks provide even further improvement over multi-class CNNs in the robust feature setting.
On CIFAR100 , in the nominal setting our greedy codebook provides an improvement of around 3% over multi-class CNN and also outperforms other codebooks as well, except 1-vs-All.
In the robust setting, both greedy and Hadamard provides high accuracy and an improvement of 2% over multi-class CNN.
On Caltech-101 , greedy codebook provides an improvement of around 0.6% over multi-class CNN in nominal setting and around 1.4% in the robust feature setting.
On Caltechour greedy codebook provides a gain of around 1% over multi-class CNN in both nominal and robust features setting; here the small gain in improvement with robust features over nominal features is consistent with the findings of [42].
It is important to note that Hadamard codes provide high accuracy only for large L , thus limiting their application to large multi-class classification problems [5].
These classification results clearly demonstrate the benefit of using ECOC-based classifiers and further highlighting the merits of our element-wise codebook design approach.
Importantly, our
IP-based Greedy algorithm provides scalability while ensuring small optimality gaps.
Further, the resulting codebooks generated, provide improved classification performance on large datasets like
CIFAR100, Caltech-101/256.
To the best of our knowledge, we are first to report such performance using ECOC-based classifiers.
Limitation: The greedy codebooks only provide a marginal improvement over Hadamard and Dense codes in terms of classification accuracy on large-class datasets.
However, note that Hadamard codes only provide high accuracy when full Hadamard code of length L ∼ 2 ⌈ log 2 ( k ) ⌉ is used.
For much smaller L , Hadamard codes perform much worse than our greedy codebooks.
Besides, as shown in [16], Dense codes are not robust to adversarial perturbations, but the proposed approach is likely to provide non-trivial adversarial robustness.
Concluding Remarks, Future Work and Societal Impact
In this paper we developed a scalable approach to solve the discrete codebook design problem for large multi-class problem instances.
Our approach generates near-optimal codebooks resulting in higher classification accuracy.
As future work, we aim to extend our design approach to other settings, for example, when the cost of miss-classification is different across different class-pairs.
Our ECOC based approach is particularly well suited to handle such tasks as we can easily switch from global error-correcting property to class-pair wise error-correcting property [33] with simple modifications.
Another extension of our work would be to very large multi-class problems [5] and extreme multi-label (XML) classification problems.
We do-not expect any negative societal impact other than that of general classification systems.