Scalable design of Error-Correcting Output Codes using Discrete Optimization with Graph Coloring
Abstract
We study the problem of scalable design of Error-Correcting Output Codes (ECOC)for multi-class classiﬁcation.
Prior works on ECOC-based classiﬁers are limited tocodebooks with small number of rows (classes) or columns, and do not provide optimality guarantees for the codebook design problem.
We address these limitationsby developing a codebook design approach based on a Mixed-Integer Quadratically
Constrained Program (MIQCP).
This discrete formulation is naturally suited formaximizing the error-correction capability of ECOC-based classiﬁers and incorporates various design criteria in a ﬂexible manner.
Our solution approach is tractablein that it incrementally increases the codebook size by adding columns to maximizethe gain in error-correcting capability.
In particular, we show that the maximalgain in error-correction can be upper bounded by solving a graph-coloring problem.
As a result, we can efﬁciently generate near-optimal codebooks for very largeproblem instances.
These codebooks provide competitive multi-class classiﬁcationperformance on small class datasets such as MNIST and CIFAR10.
Moreover,by leveraging transfer-learned binary classiﬁers, we achieve better classiﬁcationperformance over transfer-learned multi-class CNNs on large class datasets such as
CIFAR100, Caltech-101/256.
Our results highlight the advantages of simple andmodular ECOC-based classiﬁers in improving classiﬁcation accuracy without therisk of overﬁtting.
Introduction
Error-correcting codes have found many applications in machine learning.
Their use range fromnatural classiﬁcation tasks [6, 16, 2, 12] including multi-label classiﬁcation [8] to robust classiﬁcation[30, 32, 11] to federated learning [15] to zero-shot learning [27] to life-long learning [14].
Theseminal paper of [6] proposed the use of error-correcting output codes (ECOCs) for classiﬁcationtasks.
Their framework lead to a signiﬁcant activity on the design of ECOCs with a variety of features;for e.g. the use of different class of encodings [5, 12, 26], hierarchical classiﬁers [10, 9], decodingschemes [2, 7] and accounting for the underlying data-distribution [34, 33, 19, 21].
However, a keychallenge that remains to be addressed is a ﬂexible and tractable approach for designing high-qualitycodebooks.
Our paper addresses this challenge.
We tackle some of the limitations of prior works on solving the discrete codebook design problem.
In particular, we depart from the classical approach that performs a continuous relaxation of thediscrete codebook design problem and attempts to solve the resulting non-linear optimization problem[5, 34, 33, 21].
That approach has been shown to scale to large number of classes k , but under thelimitation that the size of the codebook (i.e. number of columns L ) is small.
However, with small
L , codebooks do not fully exploit the error-correcting capability, which is the key motivation to use
ECOCs in the ﬁrst place.
Besides, the continuous relaxation approach typically does not providecodebooks with strong optimality guarantees.
We formulate the codebook design problem as a mixed integer quadratically constrained program(MIQCP) and develop a greedy algorithm to efﬁciently solve this MIQCP.
Each iteration of ouralgorithm adds a new set of columns to the current codebook such that the gain in error-correctingcapability is maximal.
Importantly, we exploit a graph-coloring based upper-bound to accelerate theiterations of the greedy algorithm.
Thus, our algorithm provides the ﬂexibility to choose codebook size based on the desired level of ﬁnal accuracy.
The key advantage of our approach is that it easilyscales to large problem instances ( k = 500 ).
In comparison, the recent paper of [11] generateshigh-quality codebooks but only for k ≤ 50 .
Thus, our approach can be maximally leverage thebeneﬁt of ECOCs in aforementioned applications of error-correcting codes for realistic probleminstances.
Our main contributions are as follows: (i)
In contrast to [6, 21, 34, 33, 11], we model the codebookdesign problem as an element-wise optimization problem (MIQCP formulation).
(ii) We develop asolution approach using a greedy algorithm to solve large instances of the MIQCP with low-optimalitygaps.
(iii)
We make a theoretical connection to the vertex-coloring problem to upper bound themaximal gain in the error-correction which leads to faster iterations of the greedy algorithm.
(iv) Ourcodebooks achieve very high classiﬁcation accuracy similar to multi-class CNNs on datasets withsmall number of classes.
(v) Finally, we show a novel beneﬁt of ECOCs with transfer-learning onlarge class datasets.
Our codebooks outperform multi-class CNNs when trained with transfer-learningunder both nominal and robust feature settings.
ECOC for Classiﬁcation
The ECOC-based framework [6], encodes each class of a k -class classiﬁcation problem with a uniquecodeword of length L .
This encoding results in a codebook (coding matrix)
M of size k × L .
Forbinary codes: M ∈ { +1 , − 1 } k × L .
The rows (resp.
columns) of M correspond to distinct classes(resp.
binary classiﬁers or hypotheses ).
Figure 1 shows two example codebooks for a classiﬁcationproblem with 5 classes.
For every column in M a binary classiﬁer is trained over the training data,where all training data from classes with entry +1 (resp.
entry − 1 ) forms the positive class (resp.
theother class).
f [Equation]  fff C+ - - - -1 C-
+ - - -1 C- - + - {+-
+} C- - - + -1 C- - - - + ffffff 6 C+ + + + + +1 C- - - - + {++
+} C- - + - - -1 C-
+ + - - +
C+ - + - + -
For a test example x , let f 1 ( x ) , . .
, f L ( x ) denote the predicted class of each of the learned binaryhypotheses, where f j ( x ) ∈ { +1 , − 1 } ∀ j ∈ { 1 , . . .
, L } .
This encoding F ( x ) , where F ( x ) =[ f
1 ( x ) , . . .
, f L ( x )] , is then associated with a class (i.e., a row of coding matrix M ), typically using adecoding scheme based on a similarity measure such as Hamming distance.
A common approach is tocompute the Hamming distances d H ( · , · ) between F ( x ) and
each codeword M ( i, · ) ∀ i ∈ { 1 , . . .
, k }and predict the output class ˆ y as the class with the minimum distance:
[Equation]
F ( x )) . Importantly, the ﬁnal classiﬁcation performance of the above ECOC procedure depends on both theaccuracy of the trained binary classiﬁers and the error-correcting capability of the codebook M .
Inparticular this capability increases with the Hamming distances between rows; i.e. a codebook withhigh pairwise Hamming distances across its rows has a higher error-correction capability.
Deﬁnition 1.
Hamming distance of a codebook (denoted as κ M ) is deﬁned as the minimum hammingdistance between any two distinct pair of codewords (or rows) in M .
[Equation]
Proposition 1 (Error-Correction Capability) .
A codebook M with hamming distance κ M , can alwayscorrect at-least (cid:98) κ M − 1(cid:99) errors.
Proof of proposition 1 and all subsequent technical results are provided in supplementary information(SI).
Problem Formulation
For a k -class classiﬁcation problem we want to ﬁnd an optimal codebook of length L .
Thus, we wantto ﬁnd the value of k × L entries of a codebook, where each ( i, j ) entry (denoted as x ij ) can takevalues in { +1 , − 1 } such that the resulting codebook has high multi-class classiﬁcation accuracy.
Toachieve high classiﬁcation accuracy, we take the following design criteria [6, 11] into consideration:
The error-correcting property increases with the Hamming distance between rows; thereforeit is desirable to have a high Hamming distance between any pair of rows.
We ensure this bymaximizing - the Hamming distance of the codebook (or the minimum of the row pairwiseHamming distances).
For a binary codebook of size k × L , an analytical upper bound on thisobjective is given by the well-known Plotkin’s bound [25]: (cid:106)
Lk 2( k − 1) (cid:107) .
The Hamming distance between any two pair of columns of the codebook must be atleast 1to avoid same columns and atmost k − 1 to avoid complementary columns.
A good columnseparation is desirable in order to avoid correlations between the resulting hypotheses (or binaryclassiﬁers).
Due to these reasons, we constrain the Hamming distances between any distinct pairof columns between ρ 1 and ρ 2 , where ρ 1 ≥ 1 and ρ 2 ≤ k − 1 .
Finally, every column in the codebook should result in a balanced binary classiﬁcation problem,
i
e. each of the two classes in the binary classiﬁer should have similar number of trainingdata-points.
This automatically ensures that no column has all entries as +1 or − 1 .
For notational convenience we denote the set of row indices as N , the set of column indices T , set of 100 all pair of rows (or corresponding classes)
P N ; and the set of all pair of columns P T .
Mathematically:
N := (cid:8) 1 , . .
. , k (cid:9) , P N := (cid:110)
[Equation]
We can now formulate the element-wise codebook design problem based on the above design criteria 102 as the following Integer program ( IP 1 ):
[Equation]
The objective of IP 1 is to maximize the minimum Hamming distances between different pairsof rows (1a).
Constraint (1b) computes the Hamming distance between rows; (1c) ensures thatcolumn separation lies in the desirable range [ ρ 1 , ρ 2 ] ; (1d) ensures that every column results in avalid balanced binary classiﬁcation problem, where n i is the number of training samples in i -th classof the dataset, while γ controls the degree of allowed imbalance; and ﬁnally (1e) ensures that x ijtakes integer values in { +1 , − 1 } .
The max-min objective (1a) can be simpliﬁed by introducing anauxiliary variable t , where t :=
min { d 1 , 2 H ( x ) , d 1 , 3 H ( x ) , . . .
, d k − 1 ,k H ( x ) } and adding constraintst ≤ d 1 , 2 H ( x ) , . . .
, t ≤ d k − 1 ,k H ( x ) .
Note that (1b) and (1c) contain bilinear terms and are non-convex in nature.
Thus IP 1 can be 113 categorized as a non-convex Mixed Integer Quadratic Constrained Program (MIQCP).
IP 1 cannot besolved directly as (1b) and (1c) contains bilinear terms and therefore these constraints are linearized 115 to get a linear-relaxation of IP 1 .
Each bi-linear term is replaced with an auxiliary variable z ijpq ,where z ijpq := x ij x pq .
Additional linear constraints known as McCormick-inequalities [22] are then 117 added to lower and upper bound z ijpq .
Further details about McCormick inequalities (including theirderivation) are provided in supplementary information (SI).
For the bilinear constraint z ijpq = x ij x pq , 119 these inequalities are given by (2a) and (2b). [Equation]
[Equation]
Also, (2a) and (2b) respectively provide convex lower and upper envelopes to the feasible set admittedby the bilinear constraint [1].
Lemma 1.
McCormick relaxation of IP 1 , denoted as MC ( IP 1) , is tight: 123 (cid:8)
[Equation]
Thanks to lemma 1, IP 1 can now be solved by solving the MILP (i.e. MC ( IP 1) ) obtained afterreplacing the bilinear terms in (1b)-(1c) with the corresponding McCormick inequalities.
Note that
IP 1 has L × |P
N | + k × |P
T | =
L × (cid:0) k 2 (cid:1) + k × (cid:0) L 2 (cid:1) number of bilinear terms.
Since L ≈ k , IP 1will have O ( k 3 ) number of bilinear terms.
However in practice, linear programming (LP)-relaxationof MC ( IP 1) can be quite loose and hence even with the tight reformulation, IP 1 cannot be used totractably solve the optimal codebook design problem (especially when L is large).
Therefore, instead of generating optimal codebooks by solving IP 1 directly in one-shot, we nextdevelop a scalable greedy solution approach to generate good-quality (near optimal) codebooks forlarge k and L .
Proposed Solution Approach
In this section we develop an iterative greedy algorithm to generate near-optimal codebooks.
Todevelop our approach, we utilize the following monotonicity result of lemma 2.
Lemma 2.
Let M and ˜ M be two binary error-correcting code of size k × l and k × ( l +˜ l ) respectively,where ˜ l ≥ 0 , such that all columns in M are also in ˜ M , i.e. M ⊆ ˜ M .
Then, the Hamming distances(recall 1) of M and ˜ M satisfy: κ M ≤ κ ˜ M , implying that the error-correcting capability of ˜ M isatleast as good as M .
This intuitive result serves as the starting point of our solution approach.
In particular, from Lemma2, we know that the error correcting capability of a codebook M of size k × l can be improvedfurther by increasing the number of columns, i.e. adding ˜ l columns.
Naturally, this improvementnot only depends on ˜ l , but also on how the entries of the new columns are selected.
Our solutionapproach progressively adds new columns to an existing codebook by solving a smaller integerprogram, denoted IP 2 ; see ﬁgure 2.
This integer program maximizes the hamming distance of theresulting codebook (i.e. its error-correcting capability), while ensuring that desired column separationfor all (i.e. new and old columns) is achieved and the resulting codebook still remains a valid binaryerror-correcting code with balanced columns.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Solve
IP
[Figure]
[Equation]
Before proceeding, we introduce new notation for mathematical convenience.
Analogous to T ,(resp.
P T ) let ˜ T denote the set of new columns (resp.
P ˜ T denote the set of pair of new columns).
Furthermore, let the entries of the new columns be denoted as ˜ x ij .
Mathematically these are deﬁnedin (3), (4) and (5) respectively.
We are ready to present IP 2 which takes a valid codebook M of sizek × l and number of
IP 2 : max ˜ x
[Equation]
The objective function (6a) maximizes the the error-correcting capability.
Constraint (6b) computesthe Hamming distance between pairs of rows (or classes) in the resulting codebook; (6c) ensuresdesired column separation between any pair of new columns; (6d) ensured desired separation betweennew columns and existing columns in input M ; (6e) ensures that new columns are balanced and (6f)ensures that the new codebook is a binary codebook.
Notice the key difference between IP 2 and IP 1 : since ˜ l (cid:28) l , the constraints (6d), which accountfor the majority of columns separation conditions in the resulting codebook are linear in ˜ x as theentries of M are known.
Further, the total number of bi-linear terms in row-separation constraints(6b) and column separation (between new columns) constraints (6c) are much smaller in number.
These features of IP 2 makes it solvable even for large k by off-the-shelf IP solvers which typicallyemploy branch-and-bound (B&B) procedure.
It turns out that we can pre-compute a good quality (potentially tight) upper-bound to IP 2 .
In section4.1, we explore two different ways to generate an upper bound to IP 2 .
Our computational resultsshow that a good upper bound can accelerate the termination of the B&B procedure for IP 2 .
Sinceour greedy algorithm 4.2 relies on solving IP 2 repeatedly to increase the error-correcting capability,the overall computational gain can be signiﬁcant.
4.1 Upper Bound to IP 2
The integer constraint (6f), ˜ x ij ∈ { +1 , − 1 } in IP 2 can be equivalently re-written as ˜ x 2 ij = 1 andthe resulting problem can be categorized as a continuous , non-convex quadratic program.
We cangenerate an upper bound by taking the Rank-1 semi-deﬁnite programming (SDP) relaxation of thisquadratic program (for details see SI).
The tightness of this relaxation is same as taking the dual ofaforementioned quadratic program (equivalently IP 2 ), which is also a SDP and is dual to the Rank-
SDP relaxation.
However, solving the SDP-relaxation can be expensive for large k even with modernﬁrst-order SDP cone solvers (for eg: SCS [24]).
More importantly our computational experimentssuggest that the SDP-relaxation does not provide a good quality upper-bound to IP 2 .
We now discuss a second procedure to generate an upper bound to IP 2 .
To begin with, note that theaddition of ˜ l columns to an existing codebook M , as done by IP 2 , can be equivalently viewed asthe appending codewords (denoted as c i ) of length ˜ l to each row i of the codebook M , see ﬁgure 3.
For binary codes there are 2 ˜ l possible distinct codewords of length ˜ l , let the set of these codewordsbe denoted as C (˜ l ) .
Essentially in solving IP 2 , we seek to choose codewords from C (˜ l ) , such thatthe codebook resulting from appending them to the existing rows of M maximally increases theerror-correcting capability.
In particular, out of all (cid:0) k 2 (cid:1) row pairs, IP 2 tries to increase the hammingdistance between row pairs which are at the minimum in the existing codebook, as these minimumhamming distance row pairs decides the error-correcting capability (recall proposition 1).
Codebook M
[Equation]
[Equation]
    where c i ∈ C (˜ l ) .
For ˜ l = 2 ,
[Equation]
[Figure]
C (2) ). Therefore adding codewords of size 2 will not result in improving the hamming distance of M .
For a given codebook M , we deﬁne the set of minimum distance row-pairs (denoted E M ) as: E M = (cid:110) ( i, j ) ∈ { 1 , . . . , k } × { 1 , . . . , k } | i < j and
( M ( i
· ) , M ( j, · )) = κ M (cid:111) (7)
For example, the minimum hamming distance between any two distinct pair of rows in the codebook 190 shown in Figure 3 is 4 , and there are multiple row pairs for which the Hamming distance is 4 .
Ingeneral, for a codebook M of size k × L , we can represent the set E M ) as an undirected graph 192 (denoted G
M ) with k -vertices, where vertex i corresponds to row i in M ∀ i ∈ { 1 , . . .
, k } .
For 193 every ( i, j ) in E
M , an edge is present between vertices i and j in G M .
We refer to graph G M as the error-correcting graph of M .
Mathematically,
G M = ( V , E M ) ,
where V = { 1 , . . .
, k }
Figure 3 illustrates a codebook M and its corresponding error-correcting graph G M .
This viewpointenables us to make two important observations:
Increasing the hamming distance of row-pairs in E M by atleast 1 through addition ofcolumns (or equivalently by appending codewords c i ) to M , effectively increases theminimum hamming distance of the entire codebook by atleast 1 .2.
Any pair of distinct codewords in the set C (˜ l ) differ by at least 1 and atmost ˜ l , i e. :
[Equation] i [Equation] ˜
[Equation]
These observations lead to the following claim in proposition 2:
Proposition 2.
For a codebook M , if there exists an assignment of codeword c i ∈ C (˜ l ) to each vertexof graph G M such that no two connected vertices receive the same codeword, then the hammingdistance of the codebook κ M can be increased by atleast 1 by adding codewords from C (˜ l ) to therows of M .
In fact the condition speciﬁed by proposition 2, can be easily veriﬁed by solving an instance ofthe graph-coloring problem on G M ,
where each codeword c i ∈ C (˜ l ) can be viewed as a uniquecolor.
Before proceeding further, we recall some useful deﬁnitions related to graph coloring.
Aproper-vertex coloring is an assignment of colors to the vertices of a graph so that no two adjacentvertices have the same color.
Deﬁnition 2 (Chromatic number) .
The chromatic number of a graph G , denoted as ξ ( G ) is theminimum number of colors required for a proper vertex coloring of the graph G .
Theorem 3.
For any binary code ˜ M resulting from adding ˜ l columns to an existing binary code M ,the following holds regarding the hamming distance (recall 1) κ ˜ M of the code ˜ M :
• If the chromatic number of the graph G M , ξ ( G M ) is greater than size of the set of possible
codewords C (˜ l ) , i.e. ξ ( G M ) >
|C (˜ l ) | , then κ ˜ M = κ M .
• In particular, for ˜ l ∈ { 1 , 2 } ∗ :
[Equation]
Corollary 4.
An upper bound to IP 2 is provided by theorem 3 or IP 2 is infeasible.
Since for a given M , theorem 3 provides a valid upper-bound to the hamming distance (or equivalentlythe error-correcting capability) of all codebooks resulting from adding ˜ l columns to M , therefore itautomatically also provides an upper-bound to IP 2 .
Further, the color-assignment can be used togenerate a (partial) feasible solution to IP 2 .
Technical details of feasible solution generation areprovided in SI.
Note that the problem of computing the chromatic number of a graph (or analogously solving thevertex-coloring problem) has been extensively studied in the literature.
This includes heuristicssuch as DSATUR [3] and RLF [18], exact backtracking algorithms [17] and more recent integerprogramming based exact methods [13, 23, 20].
We solve our graph-coloring problem by formulatingand solving it as an integer program (IP); please refer to SI for details.
In table 1, we report theaverage time needed to solve the vertex-coloring problem for various error-correcting graphs G Mwith different k .
In almost all cases we are able to solve the graph-coloring problem extremely fastwithin 1-2 seconds.
This is mainly due to the highly sparse structure of G M as large number ofnodes tend to have low or even zero-degree resulting from high diversity in row pair-wise Hammingdistances.
1: Generate ﬁrst column C 1 randomly.
2: M ← C 1 and start counter: i =3: while i <
L do 4:
Compute ξ ( G M ) .
Compute upper bound to IP 2 using ξ ( G M ) and ˜ l .
Solve IP 2 with latest M , ˜ l and upper-bound.
Update M with the solution of IP8:
i = i + ˜ l 9: end while 10: return M
[Table]
4.2 Greedy Algorithm
We are now ready to present our Greedy Approach (Algorithm 1) for generating a codebook of sizek × L which is a near-optimal feasible solution to our original codebook design problem.
Initially,we randomly pick the ﬁrst column of the codebook such that it is a valid balanced binary classiﬁer.
Then, a second column is added by solving IP 2 with ˜ l = 1 -- This provides us with a codebook ofsize k × 2 that satisﬁes all the design criteria.
With this as our current codebook, we then continue toappend more columns by again solving IP 2 .
This procedure is continued until we obtain a codebookwith L columns.
Practically, it is important to keep ˜ l reasonably small so that the solution to IP 2 in each iteration of
Algorithm 1 is obtained quickly.
In the ﬁrst few iterations, since M is small, therefore the number ofcolumn-separation constraints between the existing and new columns (6d) is small.
Thus, for large k ,i.e. k > 50 , for ﬁrst (cid:100) log 2 k (cid:101) † iterations we solve IP 2 with ˜ l = 1 and use ˜ l ≥ 2 in the subsequentiterations.
We next perform computational experiments to evaluate the performance of our greedy algorithmand subsequently evaluate the classiﬁcation performance of the codebooks generated by the greedyalgorithm.
Experiments
We use Gurobi-v9.1 as our integer programming (IP) solver and run all our experiments on a machinewith Intel Core i7-6800K CPU, 32GB RAM and 1080Ti Nvidia GPU.
We will make our codeavailable at a github repository.
We compute optimality gaps of codebooks generated by differentapproaches using Plotkin’s
Bound (PB).
The gap is deﬁned as: | PB − f best | /f best , where f best denotesthe objective function value of the solution.
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
We perform our ﬁrst set of IP experiments on small problem instances ( k < 50 ) to provide a comparison between solving IP 1 directly and our Greedy approach (Algorithm 1).
We also benchmarkagainst the recent approach of Gupta and Amin [11].
We summarize our results in Table 2a.
Noticethat for very small instances such as k = 12 and k = 16 , IP 1 is able to generate a good qualitysolution, however for slightly larger problem instances (say k = 20 ) the solution quality of IP 1deteriorates rapidly (the gap is around 50% ); and ﬁnally for k ≥ 24 , IP 1 simply fails to even generatea feasible solution.
This highlights the intractability of IP 1 to generate good quality codebooks.
In contrast, our Greedy approach provides very high quality solutions.
Note that for all cases, our greedyapproach either matches or outperforms the column-subset selection (CSS) approach of [11].
Onvery small instances i.e. k ≤ 24 , [11] provides competitive performance to our greedy approach.
Thereason behind this is because [11] provides a tight IP formulation but at the cost of having exponential
O (2 k ) number of binary variables.
Their performance deteriorates after k ≥ 24 as they have to resortto random sampling to mitigate the computational expense of O (2 k ) number of binary variables.
It is not surprising that due to this exponential complexity, [11] is not tractable beyond k > 50 , asdiscussed next.
We evaluate the performance of our Greedy approach (Algorithm 1) on large problem instances,i.e. for k ≥ 50 .
These results are reported in Table 2b.
We observe that approach of [11] doesnot even provide a feasible solution for k > 50 .
In contrast, our greedy approach generates goodquality solution for even k = 500 while maintaining low optimality gaps.
For all cases our gaps arewell below 15%.
Thus our approach is signiﬁcantly more tractable than of [11].
To the best of ourknowledge, we are the ﬁrst to achieve such scalability with low optimality-gaps for the codebookdesign problem.
5.1 Classiﬁcation Performance
We now evaluate and benchmark the classiﬁcation performance of our codebooks generated usinggreedy approach on different datasets.
For each dataset, we generate codebooks of different sizesusing the Greedy approach and compare it with other commonly used codebooks like 1-vs-All, 1-vs-1[28] and Dense/Sparse codes [2].
The procedure to generate dense/sparse codes is discussed in SI.
For every codebook we train L binary classiﬁers corresponding to the columns of the codebook.
We also benchmark against a multi-class CNN.
This multi-class CNN has the same architecture asthe binary classiﬁers except for the size of the output layer, which is 2 for binary classiﬁers and kfor multi-class CNN.
For all experiments, we report average of 5 runs.
Details regarding differentdatasets, model architectures, hyper-parameters and training are provided in SI.
MNIST & CIFAR10:
In our ﬁrst set of classiﬁcation experiments, we evaluate the performance on
MNIST and CIFAR10 datasets (both with k = 10 ).
Using Greedy approach we generate codebooksof size L = 5 and 10 .
The classiﬁcation performance of different codebooks is reported in tables3 and 4 respectively.
On MNIST, the improvement in accuracy with increase in L is small, since
MNIST is a relatively easy classiﬁcation task.
However, on CIFAR10, for which the underlyingclassiﬁcation task is much harder than MNIST, we obtain a 1 . 8% gain in accuracy as the size of thecodebook increases from L = 5 to L = 10 .
Further on CIFAR10, our compact greedy codebookseasily outperforms standard 1-vs-1 codebook which uses L = 45 binary classiﬁers by around 4 .
5%and also outperforms 1-vs-All codebook by more than 1% .
For both datasets, Dense codebookprovides higher accuracy over sparse codebook, but is not able to outperform our greedy codebooks.
In comparison to multi-class CNN, our greedy codebooks provides very similar accuracy on bothdatasets, with slightly lower accuracy on MNIST but higher on CIFAR10.
[Table]
[Table]
CIFAR100 & Caltech-101/256:
In the second set of experiments we evaluate the performance onlarge class datasets: CIFAR100, Caltech-101 and Caltech-256 with k = 100 , 101 & 257 respectively.
Recall that, for these large class datasets, the approach of [11] cannot even generate a feasiblecodebook.
Thanks to our greedy algorithm, we can easily generate and evaluate our codebooks onthese datasets.
Note that the computational expense associated with the training of L binary classiﬁersfor each codebook can be large; however this is a general limitation of ECOC-based classiﬁers.
Firstly, our work addresses this limitation by generating compact codebooks with low-optimalitygaps.
Secondly, we leverage the power of transfer learning to signiﬁcantly reduce the training time ofbinary classiﬁers.
More generally, another motivation to leverage transfer-learning is due to the factthat modern deep-learning models continue to get bigger in size by the day and thus transfer-learning
[Table]
has become even more effective in the face of limited compute and data availability.
Our next resultsindeed demonstrate the additional beneﬁt which ECOCs provide with transfer-learning to furtherimprove classiﬁcation accuracy.
For transfer-learning we use models trained on ImageNet and replace the last fully-connected layer of1000 classes with 2 classes for binary classiﬁers and k classes for multi-class classiﬁer respectively.
We then freeze the weights of the all but the last fully-connected layer.
Interestingly, authors in [29]and [31] have recently shown that adversarially trained robust models provide better generalization intarget domain over nominally trained models.
Therefore for a more comprehensive evaluation of ourapproach, we use features from two different ResNet50 models trained on ImageNet.
The ﬁrst modelhas been pre-trained nominally on the ImageNet dataset while the second model has been trainedadversarially with l 2 -norm perturbations.
More details on these individual models are provided in SI.
The ﬁnal multi-class accuracies of all the codebooks and multi-class CNN for both nominal androbust features are provided in tables 5, 6 and 7 respectively.
[Table]
[Table] [Table]
We observe that on all the three (large class) datasets, our greedy codebooks consistently outperformmulti-class CNNs.
For all cases, robust features indeed provide better accuracy over nominal featuresthus validating the hypotheses of [29, 31] in the context of ECOC-based classiﬁers.
Importantly, ourgreedy codebooks provide even further improvement over multi-class CNNs in the robust featuresetting.
On CIFAR100 , in the nominal setting our greedy codebook provides an improvement of around 3%over multi-class CNN and also outperforms other codebooks as well, except 1-vs-All which providesa slightly higher accuracy than our greedy codebook.
In the robust setting, our codebook providesan improvement of 2% over multi-class CNN and easily outperforms other codebooks including1-vs-All.
On Caltech-101 , an improvement of around 0.6% is achieved over multi-class CNN in nominal settingand around 1.4% in the robust feature setting.
Our greedy codebooks outperform other codebooks aswell.
On Caltech-256 our greedy codebooks provide a gain of around 1% over multi-class CNN in bothnominal and robust features setting; we also outperform other codebooks.
Here the small gain inimprovement with robust features over nominal features is consistent with the ﬁndings of [29].
These classiﬁcation results clearly demonstrate the beneﬁt of using ECOC-based classiﬁers andfurther highlighting the merits of our element-wise codebook design approach.
Importantly, our
IP-based Greedy algorithm provides scalability while ensuring small optimality gaps.
Further, theresulting codebooks generated, provide improved classiﬁcation performance on large datasets like
CIFAR100, Caltech-101/256.
To the best of our knowledge, we are ﬁrst to report such performanceusing ECOC-based classiﬁers.
Concluding Remarks, Future Work and Societal Impact
In this paper we developed a scalable approach to solve the discrete codebook design problem forlarge multi-class problem instances.
Our approach generates near-optimal codebooks resulting inhigher classiﬁcation accuracy.
As future work, we aim to extend our design approach to other settings,for example, when the cost of miss-classiﬁcation is different across different class-pairs.
Our ECOCbased approach is particularly well suited to handle such tasks as we can easily switch from globalerror-correcting property to class-pair wise error-correcting property [21] with simple modiﬁcations.
We do-not expect any negative societal impact other than that of general classiﬁcation systems. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See end of page 8 and beginningof page 9 and also see supplementary section.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Seesection 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Proof of all technical
results are provided in Supplementary section.
In the main text, we have tried to provide
intuition about various results along with clear images.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See beginningof section 5.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We have provided sufﬁcient details in the main text along with a
more detailed description in the Supplementary section.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report averages of 5 runs and correspondingerror-bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See beginning of section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] All the dataset in the paper used are
open source, freely available and have been appropriately cited (see supplementary
section).
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]