Original: A N I NFORMATION -T HEORETIC M ETRIC OF T RANSFERABILITY FOR T ASK T RANSFER L EARNING
Revised: A N I NFORMATION -T HEORETIC M ETRIC OF T RANSFERABILITY FOR T ASK T RANSFER L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task.
Revised: An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task.

Original: Typically, transferability is either measured experimentally or inferred through task relatedness, which is often deﬁned without a clear operational meaning.
Revised: Typically, transferability is either measured experimentally or inferred through task relatedness, which is often deﬁned without a clear operational meaning.

Original: In this paper, we present a novel metric, H -score , an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classiﬁcation problems.
Revised: In this paper, we present a novel metric, H -score , an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classiﬁcation problems.

Original: Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature.
Revised: Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature.

Original: This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efﬁcient transfer learning policies.
Revised: This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efﬁcient transfer learning policies.

Original: Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classiﬁcation, such as recognition tasks for 3D indoor-scene understanding.
Revised: Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classiﬁcation, such as recognition tasks for 3D indoor-scene understanding.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transfer learning is a learning paradigm that exploits relatedness between different learning tasks in order to gain certain beneﬁts, e.g. reducing the demand for supervision (Pratt (1993)).
Revised: Transfer learning is a learning paradigm that exploits relatedness between different learning tasks in order to gain certain beneﬁts, e.g. reducing the demand for supervision (Pratt (1993)).

Original: In task transfer learning , we assume that the input domain of the different tasks are the same.
Revised: In task transfer learning , we assume that the input domain of the different tasks are the same.

Original: Then for a target task
Revised: Then for a target task

Original: T T , instead of learning a model from scratch, we can initialize the parameters from a previously trained model for some related source task T S .
Revised: T T , instead of learning a model from scratch, we can initialize the parameters from a previously trained model for some related source task T S .

Original: For example, deep convolutional neural networks trained for the ImageNet classiﬁcation task have been used as the source network in transfer learning for target tasks with fewer labeled data (Donahue et al.
Revised: For example, deep convolutional neural networks trained for the ImageNet classiﬁcation task have been used as the source network in transfer learning for target tasks with fewer labeled data (Donahue et al.

Original: (2014)), such as medical image analysis (Shie et al.
Revised: (2014)), such as medical image analysis (Shie et al.

Original: and structural damage recognition in buildings (Gao & Mosalam).
Revised: and structural damage recognition in buildings (Gao & Mosalam).

Original: An imperative question in task transfer learning is transferability , i.e. when a transfer may work and to what extent .
Revised: An imperative question in task transfer learning is transferability , i.e. when a transfer may work and to what extent .

Original: Given a metric capable of efﬁciently and accurately measuring transferability across arbitrary tasks, the problem of task transfer learning, to a large extent, is simpliﬁed to search procedures over potential transfer sources and targets as quantiﬁed by the metric.
Revised: Given a metric capable of efﬁciently and accurately measuring transferability across arbitrary tasks, the problem of task transfer learning, to a large extent, is simpliﬁed to search procedures over potential transfer sources and targets as quantiﬁed by the metric.

Original: Traditionally, transferability is measured purely empirically using model loss or accuracy on the validation set (Yosinski et al.
Revised: Traditionally, transferability is measured purely empirically using model loss or accuracy on the validation set (Yosinski et al.

Original: (2014); Zamir et al.
Revised: (2014); Zamir et al.

Original: ; Conneau et al.
Revised: ; Conneau et al.

Original: There have been theoretical studies that focus on task relatedness (Baxter (2000); Maurer (2009); Pentina & Lampert (2014); Ben-David et al.
Revised: There have been theoretical studies that focus on task relatedness (Baxter (2000); Maurer (2009); Pentina & Lampert (2014); Ben-David et al.

Original: However, they either cannot be computed explicitly from data or do not directly explain task transfer performance.
Revised: However, they either cannot be computed explicitly from data or do not directly explain task transfer performance.

Original: In this study, we aim to estimate transferability analytically, directly from the training data .
Revised: In this study, we aim to estimate transferability analytically, directly from the training data .

Original: We quantify the transferability of feature representations across tasks via an approach grounded in statistics and information theory.
Revised: We quantify the transferability of feature representations across tasks via an approach grounded in statistics and information theory.

Original: The key idea of our method is to show that the error probability of using a feature of the input data to solve a learning task can be characterized by a linear projection of this feature between the input and output domains.
Revised: The key idea of our method is to show that the error probability of using a feature of the input data to solve a learning task can be characterized by a linear projection of this feature between the input and output domains.

Original: Hence we adopt the projection length as a metric of the feature’s effectiveness for the given task, and refer to it as the H-score of the feature.
Revised: Hence we adopt the projection length as a metric of the feature’s effectiveness for the given task, and refer to it as the H-score of the feature.

Original: More generally, H-score can be applied to evaluate the performance of features in different tasks, and is particularly useful to quantify feature transferability among tasks.
Revised: More generally, H-score can be applied to evaluate the performance of features in different tasks, and is particularly useful to quantify feature transferability among tasks.

Original: Using this idea, we deﬁne task transferability as the normalized H-score of the optimal source feature with respect to the target task.
Revised: Using this idea, we deﬁne task transferability as the normalized H-score of the optimal source feature with respect to the target task.

Original: As we demonstrate in this paper, the advantage of our transferability metric is threefold.
Revised: As we demonstrate in this paper, the advantage of our transferability metric is threefold.

Original: (i) it has a strong operational meaning rooted in statistics and information theory; (ii) it can be computed directly and efﬁciently from the input data, with fewer samples than those needed for empirical learning; (iii) it can be shown to be strongly consistent with empirical transferability measurements.
Revised: (i) it has a strong operational meaning rooted in statistics and information theory; (ii) it can be computed directly and efﬁciently from the input data, with fewer samples than those needed for empirical learning; (iii) it can be shown to be strongly consistent with empirical transferability measurements.

Original: In this paper, we will ﬁrst present the theoretical results of the proposed transferability metric in
Revised: In this paper, we will ﬁrst present the theoretical results of the proposed transferability metric in

Original: Section 2-4.
Revised: Section 2-4.

Original: Section 5 presents several experiments on real image data , including image classiﬁcaton tasks using the Cifar 100 dataset and 3D indoor scene understanding tasks using the Taskonomy dataset created by Zamir et al.
Revised: Section 5 presents several experiments on real image data , including image classiﬁcaton tasks using the Cifar 100 dataset and 3D indoor scene understanding tasks using the Taskonomy dataset created by Zamir et al.

Original: A brief review of the related works is included in Section 6.
Revised: A brief review of the related works is included in Section 6.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we will introduce the notations used throughout this paper, as well as some related concepts in Euclidean information theory and statistics.
Revised: In this section, we will introduce the notations used throughout this paper, as well as some related concepts in Euclidean information theory and statistics.

Original: X , x , X and P X represent a random variable, a value, the alphabet and the probability distribution respectively.
Revised: X , x , X and P X represent a random variable, a value, the alphabet and the probability distribution respectively.

Original: √ P X denotes the vector with entries (cid:112) P X ( x ) and [ √ P X ] the diagonal matrix of (cid:112)
Revised: √ P X denotes the vector with entries (cid:112) P X ( x ) and [ √ P X ] the diagonal matrix of (cid:112)

Original: P X ( X ) .
Revised: P X ( X ) .

Original: For joint distribution P Y X , P Y X represents the |Y| × |X| probability matrix.
Revised: For joint distribution P Y X , P Y X represents the |Y| × |X| probability matrix.

Original: Depending on the context, f( X ) is either a |X| -dimensional vector whose entries are f ( x ) , or a |X| × k feature matrix.
Revised: Depending on the context, f( X ) is either a |X| -dimensional vector whose entries are f ( x ) , or a |X| × k feature matrix.

Original: Further, we deﬁne a task to be a tuple T =
Revised: Further, we deﬁne a task to be a tuple T =

Original: ( X, Y, P XY ) , where X is the training features and Y is the training label, and P XY the joint probability (possibly unknown).
Revised: ( X, Y, P XY ) , where X is the training features and Y is the training label, and P XY the joint probability (possibly unknown).

Original: Subscripts S and T are used to distinguish the source task from the target task.
Revised: Subscripts S and T are used to distinguish the source task from the target task.

Original: 2. L OCAL I NFORMATION G EOMETRY AND H YPOTHESIS T ESTING
Revised: 2. L OCAL I NFORMATION G EOMETRY AND H YPOTHESIS T ESTING

Original: Our deﬁniton of transferability uses concepts in local information geometry developed by Makur et al.
Revised: Our deﬁniton of transferability uses concepts in local information geometry developed by Makur et al.

Original: (2015), which characterizes probability distributions as vectors in the information space.
Revised: (2015), which characterizes probability distributions as vectors in the information space.

Original: Consider the following binary hypothesis testing problem: test whether i.i.d.
Revised: Consider the following binary hypothesis testing problem: test whether i.i.d.

Original: samples  { x ( i ) } mi =1 are drawn from distribution P 1 or distribution P 2 , where P 1 , P 2 belong to an (cid:15) -neighborhood N (cid:15) ( P 0 )
Revised: samples x m = { x ( i ) } mi =1 are drawn from distribution P 1 or distribution P 2 , where P 1 , P 2 belong to an (cid:15) -neighborhood N (cid:15) ( P 0 )

Original: (cid:44) { P | (cid:80) x ∈X ( P ( x ) − P 0 ( x )) 2 P [Equation] x ) ≤ (cid:15) 2 } centered at a reference distributon P 0 .
Revised: (cid:44) { P | (cid:80) x ∈X ( P ( x ) − P 0 ( x )) 2 P 0 ( x ) ≤ (cid:15) 2 } centered at a reference distributon P 0 .

Original: And denote φ i
Revised: And denote φ i

Original: = P i ( x ) − P 0 ( x ) (cid:15) √ P 0 ( x ) as the information vector corresponding to P i for i = 1 , 2 .
Revised: = P i ( x ) − P 0 ( x ) (cid:15) √ P 0 ( x ) as the information vector corresponding to P i for i = 1 , 2 .

Original: Given k normalized feature functions f ( x ) =
Revised: Given k normalized feature functions f ( x ) =

Original: [ f 1 ( x ) , . .
Revised: [ f 1 ( x ) , . .

Original: , f k ( x )] , let Ξ( x ) =
Revised: , f k ( x )] , let Ξ( x ) =

Original: [ ξ 1 ( x ) , . . . , ξ k ( x )] be the matrix of feature vectors ξ i ( x ) =
Revised: [ ξ 1 ( x ) , . . . , ξ k ( x )] be the matrix of feature vectors ξ i ( x ) =

Original: (cid:112) P 0 ( x ) f i ( x ) .
Revised: (cid:112) P 0 ( x ) f i ( x ) .

Original: It can beshown that the error exponentcorresponding to f ( x  )  can be written as the squared length of a projection:
Revised: Given observations x m , feature f is associated with a k-dimensional statistics l ( x m ) = 1 m (cid:80) mi =1 ( f ( x ( i ) )) for the binary hypothesis testing problem. Let E f be the error exponent of decision region { x m | l ( x m ) > T } for T ≥ 0 , which characterizes the asymptotic error probability P e of l (i.e. lim m →∞ − 1 m log( P e ) = E kf ). E f can be written as the squared length of a projection:

Original: [Equation]
Revised: [Equation]

Revised: When f ( x ) = log P 1 ( x ) P 2 ( x ) is the log likelihood ratio, l is the minimum sufﬁcient statistics that achieves the largest error exponent E kf

Original: {+= (cid:80) ki =1 (cid:15) 2 8 || φ 1 − φ 2 || 
+}
Revised: {+= (cid:80) ki =1 (cid:15) 2 8 || φ 1 − φ 2 || 
+}

Revised: + o ( (cid:15) 2 ) by the Chernoff theorem.

Original: (See Appendix A for details.)
Revised: (See Appendix A for details.)

Original: In the rest of this paper, we assume (cid:15) is small.
Revised: In the rest of this paper, we assume (cid:15) is small.

Original: 2. D IVERGENCE T RANSITION M ATRIX
Revised: 2. D IVERGENCE T RANSITION M ATRIX

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Matrix ˜ B is the Divergence Transition Matrix (DTM) of a joint probability P Y X if ˜ B = (cid:2) √ P Y (cid:3) − 1 P Y X (cid:2) √ P X (cid:3) − 1 − √ P Y √ P X T .
Revised: Matrix ˜ B is the Divergence Transition Matrix (DTM) of a joint probability P Y X if ˜ B = (cid:2) √ P Y (cid:3) − 1 P Y X (cid:2) √ P X (cid:3) − 1 − √ P Y √ P X T .

Original: The singular values of ˜ B satisfy that 1 ≥ σ 1 ≥ · · · ≥ σ K = 0 , K (cid:44) min {|X| , |Y|} .
Revised: The singular values of ˜ B satisfy that 1 ≥ σ 1 ≥ · · · ≥ σ K = 0 , K (cid:44) min {|X| , |Y|} .

Original: Let Ψ =
Revised: Let Ψ =

Original: [ ψ 1 , . . .
Revised: [ ψ 1 , . . .

Original: , ψ K ] and Φ =
Revised: , ψ K ] and Φ =

Original: [ φ 1 , . .
Revised: [ φ 1 , . .

Original: . , φ K ] be the left and right singular vectors of ˜ B .
Revised: . , φ K ] be the left and right singular vectors of ˜ B .

Original: Deﬁne functions f ∗ i ( x )
Revised: Deﬁne functions f ∗ i ( x )

Original: =
Revised: =

Original: φ i ( x ) √ P X ( x ) and g ∗ i ( y )
Revised: φ i ( x ) √ P X ( x ) and g ∗ i ( y )

Original: =
Revised: =

Original: ψ i ( y ) √ P Y ( y ) for each i = 1 , . . .
Revised: ψ i ( y ) √ P Y ( y ) for each i = 1 , . . .

Original: , K − 1 .
Revised: , K − 1 .

Original: Makur et al.
Revised: Makur et al.

Original: (2015)  proved that f ∗ i and g ∗ i are solutions to the maximal HGR correlation problem studied by Hirschfeld (1935);
Revised: (2015) further proved that f ∗ i and g ∗ i are solutions to the maximal HGR correlation problem studied by Hirschfeld (1935);

Original: Gebelein (1941); R´enyi (1959), which ﬁnds the K strongest, independent modes in P XY from data.
Revised: Gebelein (1941); R´enyi (1959), deﬁned as follows: [Equation] The maximal HGR problem ﬁnds the K strongest, independent modes in P XY from data.

Revised: It can be solved efﬁciently using the Alternating Conditional Expectation (ACE) algorithm with provable error bound (see Appendix B).

Original: Huang et al.
Revised: Huang et al.

Original: (2017) further showed that they are the universal minimum error probability features in the sense that they can achieve the smallest error probability over all possible inference tasks.
Revised: (2017) further showed that f ∗ and g ∗ are the universal minimum error probability features in the sense that they can achieve the smallest error probability over all possible inference tasks.

Original:  M EASURING THE E FFECTIVENESS OF F EATURES IN C LASSIFICATION T ASKS
Revised:  M EASURING THE E FFECTIVENESS OF F EATURES IN C LASSIFICATION T ASKS

Original: In this section, we present a performance metric of a given feature representation for a learning task.
Revised: In this section, we present a performance metric of a given feature representation for a learning task.

Original: 3. H-S CORE
Revised: 3. H-S CORE

Original: For a classiﬁcation task involving input variable X and label Y , most learning algorithms work by ﬁnding a k -dimensional functional representation f ( x ) of X that is most discriminative for the classiﬁcation.
Revised: For a classiﬁcation task involving input variable X and label Y , most learning algorithms work by ﬁnding a k -dimensional functional representation f ( x ) of X that is most discriminative for the classiﬁcation.

Original: To measure how effective f ( x ) is in predicting Y , rather than train the model via gradient descent and evaluate its accuracy, we present an analytical approach based on the deﬁnition below:
Revised: To measure how effective f ( x ) is in predicting Y , rather than train the model via gradient descent and evaluate its accuracy, we present an analytical approach based on the deﬁnition below:

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Given data matrix X ∈ R m × d and label Y ∈ { 1 , . . .
Revised: Given data matrix X ∈ R m × d and label Y ∈ { 1 , . . .

Original: , |Y|} .
Revised: , |Y|} .

Original: Let f ( x ) be a kdimensional, zero-mean feature function.
Revised: Let f ( x ) be a kdimensional, zero-mean feature function.

Original: The H-Score of f ( x ) with respect to the learning task represented by P Y X is:
Revised: The H-Score of f ( x ) with respect to the learning task represented by P Y X is:

Original: [Equation]
Revised: [Equation]

Original: This deﬁnition is intuitive from a nearest neighbor search perspective.
Revised: This deﬁnition is intuitive from a nearest neighbor search perspective.

Original: i.e. a high H-score implies the inter-class variance cov ( E P X | Y [ f ( X ) | Y ]) of f is large, while feature redundancy tr ( cov ( f ( X ))) is small.
Revised: i.e. a high H-score implies the inter-class variance cov ( E P X | Y [ f ( X ) | Y ]) of f is large, while feature redundancy tr ( cov ( f ( X ))) is small.

Original: Such a feature is discriminative and efﬁcient for learning label Y. More importantly, H ( f ) has a deeper operational meaning related to the asymptotic error probability for a decision rule based on f in the hypothesis testing context, discussed in the next section.
Revised: Such a feature is discriminative and efﬁcient for learning label Y. More importantly, H ( f ) has a deeper operational meaning related to the asymptotic error probability for a decision rule based on f in the hypothesis testing context, discussed in the next section.

Original: 3. O PERATIONAL M EANING OF H-S CORE
Revised: 3. O PERATIONAL M EANING OF H-S CORE

Original: Without loss of generality, we consider the binary classiﬁcation task as a hypothesis testing problem deﬁned in Section 2.1, with P 1 = P X | Y =0 , and P 2 = P X | Y =1 .
Revised: Without loss of generality, we consider the binary classiﬁcation task as a hypothesis testing problem deﬁned in Section 2.1, with P 1 = P X | Y =0 , and P 2 = P X | Y =1 .

Original: For any k-dimensional feature representation f ( x ) , we can quantify its performance with respect to the learning task using its error exponent E kf .
Revised: For any k-dimensional feature representation f ( x ) , we can quantify its performance with respect to the learning task using its error exponent E kf .

Original: Theorem 1.
Revised: Theorem 1.

Original: Given P X | Y =0 , P X | Y =1 ∈ N X (cid:15) ( P 0 ,X ) and features f such that E [ f ( X )] = 0 and
Revised: Given P X | Y =0 , P X | Y =1 ∈ N X (cid:15) ( P 0 ,X ) and features f such that E [ f ( X )] = 0 and

Original: E [ f ( X ) f
Revised: E [ f ( X ) f

Original: ( X ) T ]
Revised: ( X ) T ]

Original: =
Revised: =

Original: I , there exists some constant c independent of f such that E kf =
Revised: I , there exists some constant c independent of f such that E kf =

Original: c H ( f ) .
Revised: c H ( f ) .

Original: See Appendix B for the proof.
Revised: See Appendix C for the proof.

Original: The above theorem shows that H-score H ( f ) is proportional to the error exponent of the decision function based on f ( x ) when f ( x ) is zero-mean with identity covariance.
Revised: The above theorem shows that H-score H ( f ) is proportional to the error exponent of the decision region based on f ( x ) when f ( x ) is zero-mean with identity covariance.

Original: To compute the H-score of arbitrary f , we can center the features f S ( x ) − E [ f S ( x )] , and incorporate normalization into the computation of the error exponent, which results in Deﬁnition 2.
Revised: To compute the H-score of arbitrary f , we can center the features f S ( x ) − E [ f S ( x )] , and incorporate normalization into the computation of the error exponent, which results in Deﬁnition 2.

Original: The details are presented in Appendix C.
Revised: The details are presented in Appendix D.

Original: The proof for Theorem 1 uses the fact that H ( f ) = (cid:107) ˜ B Ξ (cid:107) 2 F , where ˜ B is the DTM matrix, Ξ (cid:44) [ ξ 1 · · · ξ k ] is the matrix composed of information vectors ξ i and c is a constant.
Revised: The proof for Theorem 1 uses the fact that H ( f ) = (cid:107) ˜ B Ξ (cid:107) 2 F , where ˜ B is the DTM matrix, Ξ (cid:44) [ ξ 1 · · · ξ k ] is the matrix composed of information vectors ξ i and c is a constant.

Original: This allows us to infer an upper bound for the H-score of a given learning task:
Revised: This allows us to infer an upper bound for the H-score of a given learning task:

Original: Corollary 1.
Revised: Corollary 1.

Original: For all normalized feature f ( x ) =
Revised: For all normalized feature f ( x ) =

Original: [Equation] [Equation]
Revised: [ f 1 ( x ) , . .

Original: ] , its H-score satisﬁes [Equation] ki = [Equation]
Revised: , f k ( x )] , its H-score satisﬁes H ( f ) ≤ (cid:80) ki =1 σ 2 i ≤ k , where σ i is the ith singular value of ˜ B .

Original: i

Original: [Equation]
Revised: [Figure]

Original: k , where σ i is the ith singular value of ˜ B .

Original: The ﬁrst inequality is achieved when Ξ is composed of the right singular vectors of ˜ B , i.e. max Ξ || ˜ B Ξ || 2 F = ||
Revised: The ﬁrst inequality is achieved when Ξ is composed of the right singular vectors of ˜ B , i.e. max Ξ || ˜ B Ξ || 2 F = ||

Original: ˜ B T || 2 F .
Revised: ˜ B T || 2 F .

Original: The corresponding feature functions f opt ( X ) is in fact the same as the universal minimum error probability features from the maximum HGR correlation problem.
Revised: The corresponding feature functions f opt ( X ) is in fact the same as the universal minimum error probability features from the maximum HGR correlation problem.

Original: The ﬁnal inequality in Corollary 1 is due to the fact all singular values of ˜ B are less than or equal to 1.
Revised: The ﬁnal inequality in Corollary 1 is due to the fact all singular values of ˜ B are less than or equal to 1.

Original:  T RANSFERABILITY
Revised:  T RANSFERABILITY

Original: Next, we apply H-score to efﬁciently measure the effectiveness of task transfer learning.
Revised: Next, we apply H-score to efﬁciently measure the effectiveness of task transfer learning.

Original: We will also discuss how this approach can be used to solve the source task selection problem.
Revised: We will also discuss how this approach can be used to solve the source task selection problem.

Original: 4. T ASK TRANSFERABILITY
Revised: 4. T ASK TRANSFERABILITY

Original: A typical way to transfer knowledge from the source task T S to target task T T is to train the target task using source feature representation f S ( x ) .
Revised: A typical way to transfer knowledge from the source task T S to target task T T is to train the target task using source feature representation f S ( x ) .

Original: In a neural network setting, this idea can be implemented[Figure] by copying the parameters from the ﬁrst N layers in the trained source model to the target model, assuming the model architecture on those layers are the same.
Revised: In a neural network setting, this idea can be implemented by copying the parameters from the ﬁrst N layers in the trained source model to the target model, assuming the model architecture on those layers are the same.

Original: The target classiﬁer then can be trained while freezing parameters in the copied layers (Figure 1).
Revised: The target classiﬁer then can be trained while freezing parameters in the copied layers (Figure 1).

Original: Under this model, a natural way to quantify transferability is as follows:
Revised: Under this model, a natural way to quantify transferability is as follows:

Original: Deﬁnition 3 (Task transferability) .
Revised: Deﬁnition 3 (Task transferability) .

Original: Given source task T S and target task T T , and trained source feature representation f S ( x ) , the transferability from T S to T T is T ( S, T ) (cid:44) H T ( f S ) H T ( f T opt ) , where f T opt ( x ) is the minimum error probability feature of the target task.
Revised: Given source task T S and target task T T , and trained source feature representation f S ( x ) , the transferability from T S to T T is T ( S, T ) (cid:44) H T ( f S ) H T ( f T opt ) , where f T opt ( x ) is the minimum error probability feature of the target task.

Original: The statement T ( S, T ) =
Revised: The statement T ( S, T ) =

Original: r means the error exponent of transfering from T S via feature representation f S is 1 r of the optimal error exponent for predicting the target label Y T .
Revised: r means the error exponent of transfering from T S via feature representation f S is 1 r of the optimal error exponent for predicting the target label Y T .

Original: This deﬁnition also implies 0 ≤ T ( S, T ) ≤ 1 , which satisﬁes the data processing inequality if we consider the transferred feature f S ( X ) as post-processing of input X for solving the target task.
Revised: This deﬁnition also implies 0 ≤ T ( S, T ) ≤ 1 , which satisﬁes the data processing inequality if we consider the transferred feature f S ( X ) as post-processing of input X for solving the target task.

Original: And it can not increase information about predicting the target task T .
Revised: And it can not increase information about predicting the target task T .

Original: A common technique in task transfer learning is ﬁne-tuning, which adds before the target classiﬁer additional free layers, whose parameters are optimized with respect to the target label.
Revised: A common technique in task transfer learning is ﬁne-tuning, which adds before the target classiﬁer additional free layers, whose parameters are optimized with respect to the target label.

Original: For the operational meaning of transferability to hold exactly, we require the ﬁne tuning layers consist of only linear transformations, such as the model illustrated in Figure 2.
Revised: For the operational meaning of transferability to hold exactly, we require the ﬁne tuning layers consist of only linear transformations, such as the model illustrated in Figure 2.a.

Original: It can be shown that under the local assumption, H-score is equivalent to the log-loss of the linear transfer model up to a constant offset (Appendix D).
Revised: It can be shown that under the local assumption, H-score is equivalent to the log-loss of the linear transfer model up to a constant offset (Appendix E).

Original: Nevertheless, later we will demonstrate empirically that this transferability metric can still be used for comparing the relative task transferability with ﬁne-tuning.
Revised: Nevertheless, later we will demonstrate empirically that this transferability metric can still be used for comparing the relative task transferability with ﬁne-tuning.

Original: 4. E FFICIENT C OMPUTATION OF T RANSFERABILITY
Revised: 4. E FFICIENT C OMPUTATION OF T RANSFERABILITY

Original: With a known f , computing H-score from m sample data only takes O ( mk 2 ) time, where k is the dimension of f ( x ) for k < m .
Revised: With a known f , computing H-score from m sample data only takes O ( mk 2 ) time, where k is the dimension of f ( x ) for k < m .

Original: The majority of the computation time is spent on computing the sample covariance matrix cov ( f ( X )) .
Revised: The majority of the computation time is spent on computing the sample covariance matrix cov ( f ( X )) .

Original: The remaining question is how to obtain f T opt efﬁciently.
Revised: The remaining question is how to obtain f T opt efﬁciently.

Original: We use the fact that H T ( f opt ) = ||
Revised: We use the fact that H T ( f opt ) = ||

Original: ˜ B T || 2 F =
Revised: ˜ B T || 2 F =

Original: E [ f ( X ) T g ( Y )] , where f and g are the solutions of the HGR-Maximum Correlation problem.
Revised: E [ f ( X ) T g ( Y )] , where f and g are the solutions of the HGR-Maximum Correlation problem.

Original: This problem can be solved efﬁciently using the Alternative Conditional Expectation (ACE) algorithm fora discrete alphabet X (Makur et al.
Revised: This problem can be solved efﬁciently using the ACE algorithm for discrete variable X . For a continuous random variable X , we can obtain f opt through a different formulation of the HGR maximal W Y T f S ( x )

Revised: [Figure]

Revised: X

Original: (2015)), or equivalently by solving the following problem for a continuous scenario:
Revised: Y s source [Figure] target task f S ( x ) W g s ( y ) g s ( y ) X

Revised: Y f ( x ) f encoder g encoder g ( y ) ! "

Revised: # $ % & −2*+(-./(" # -./ % &

Revised: L

Revised: [Figure]

Revised: 6 dim(f(x)) 0.

Revised: H-score: H Y ( f ( x ))

Revised: 2 I ( X ; Y ) correlation problem:

Original: [Equation]
Revised: [Equation]

Revised: This is also known as the soft HGR problem studied by Wang et al.

Revised: (2018), who reformulated the original maximal HGR correlation objective to eliminate the whitening constraints while having theoretically equivalent solution.

Revised: In practice, we can utilize neural network layers to model functions f and g , as shown in Figure 2.b.

Revised: Given two branches of k output units for both f and g , the loss function can be evaluated in O ( mk 2 ) , where m is the batch size.

Original: Makur et al.
Revised: Makur et al.

Original: (2015) showd that the sample complexity of ACE is only 1 /k of the complexity of estimating P Y,X directly.
Revised: (2015) showed that the sample complexity of ACE is only 1 /k of the complexity of estimating P Y,X directly.

Revised: This result also applies to the soft HGR problem due to their theoretical equivalence.

Original: Hence transferability can be computed with much less samples than actually training the transfer network.
Revised: Hence transferability can be computed with much less samples than actually training the transfer network.

Revised: It’s also worth noting that, when f is ﬁxed, maximizing the objective in Equation 3 with respect to zero-mean function g results in the deﬁnition of H-score.

Original: In many cases though, the computation of H T ( f opt ) can even be skipped entirely, such as the problem below:
Revised: In many cases though, the computation of H T ( f opt ) can even be skipped entirely, such as the problem below:

Original: Deﬁnition 4 (Source task selection) .
Revised: Deﬁnition 4 (Source task selection) .

Original: Given N
Revised: Given N

Original: source
Revised: source

Original: tasks T S 1 , . .
Revised: tasks T S 1 , . .

Original: , T S N with labels Y S 1 , . . .
Revised: , T S N with labels Y S 1 , . . .

Original: , Y S N and a target task T T with label Y T .
Revised: , Y S N and a target task T T with label Y T .

Original: Let f S 1 , . .
Revised: Let f S 1 , . .

Original: . , f S N be the minimum error probability feature functions of the source tasks.
Revised: . , f S N be the minimum error probability feature functions of the source tasks.

Original: Find the source task T S i that maximizes the testing accuracy of predicting Y T with feature f S i .[Figure]
Revised: Find the source task T S i that maximizes the testing accuracy of predicting Y T with feature f S i .

Original: [Figure]

Original: We can solve this problem by selecting the source task with the largest transferability to T T .
Revised: We can solve this problem by selecting the source task with the largest transferability to T T .

Original: In fact, we only need to compute the numerator in the transferability deﬁnition since the denominator is the same for all source tasks, i.e. argmax i T ( S i , T ) =
Revised: In fact, we only need to compute the numerator in the transferability deﬁnition since the denominator is the same for all source tasks, i.e. argmax i T ( S i , T ) =

Original: argmax i ( H T ( f S i, opt ) /
Revised: argmax i ( H T ( f S i, opt ) /

Original: H T ( f T opt ) =
Revised: H T ( f T opt ) =

Original: argmax i H ( f S i ) .
Revised: argmax i H ( f S i ) .

Original: 4. R ELATIONSHIP WITH M UTUAL I NFORMATION
Revised: 4. R ELATIONSHIP WITH M UTUAL I NFORMATION

Original: Under the local assumption that P X | Y ∈ N (cid:15) ( P X ) , we can show that mutual information I ( X ; Y ) =
Revised: Under the local assumption that P X | Y ∈ N (cid:15) ( P X ) , we can show that mutual information I ( X ; Y ) =

Original: 12 || B
Revised: 12 || B

Original: || 2 F
Revised: || 2 F

Original: + o ( (cid:15) 2 ) .
Revised: + o ( (cid:15) 2 ) .

Original: (See Appendix E for details.)
Revised: (See Appendix F for details.)

Original: Hence H-score is related to mutual information by
Revised: Hence H-score is related to mutual information by

Original: H ( f ( x )) ≤ 2 I ( X ; Y ) for any zero-mean features f ( x ) satisfying the aforementioned conditions.
Revised: H ( f ( x )) ≤ 2 I ( X ; Y ) for any zero-mean features f ( x ) satisfying the aforementioned conditions.

Original: Figure 3 compares the optimal H-score of a synthesized task when |Y| = 6 with the mutual information between input and output variables, when the feature dimension k changes.
Revised: Figure 3 compares the optimal H-score of a synthesized task when |Y| = 6 with the mutual information between input and output variables, when the feature dimension k changes.

Original: The value of H-score increases as k increases, but reaches the upper bound when k ≥ 6 , since the rank of the joint probability between X and Y T , as well as the rank of its DTM is 6.
Revised: The value of H-score increases as k increases, but reaches the upper bound when k ≥ 6 , since the rank of the joint probability between X and Y T , as well as the rank of its DTM is 6.

Original: Function 2 I ( X ; Y ) , on the other hand, is constant for all k . It is slightly larger than H-score, due to the constant o ( (cid:15) 2 ) .
Revised: As expected, the H-score values are below 2 I ( X ; Y ) , with a gap due to the constant o ( (cid:15) 2 ) .

Original: We consider H-score a better transferability metric than mutual information since it is adaptive to feature dimensions, and can capture asymmetric transfer performance.
Revised: This relationship shows that H-score is consistent with mutual information with a sufﬁciently large feature dimension k . In practice, H-score is much easier to compute than mutual information when the input variable

Original: For example, in the visual recognition domain, we found it’s more effective to transfer from 3D tasks to 2D tasks than the other way around.
Revised: X (or f S ( X ) ) is continous, as mutual information are either computed based on binning, which has extra bias due to bin size, or more sophisticated methods such as kernel density estimation or neural networks (Gabri´e et al. On the other hand, H-score only needs to estimate conditional expectations, which requires less samples.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we validate and analyze our transferability metric through experiments on real image data.
Revised: In this section, we validate and analyze our transferability metric through experiments on real image data.

Original: The tasks considered cover both object classiﬁcation and non-classiﬁcation tasks in computer vision, such as depth estimation and 3D (occlusion) edge detection.
Revised: The tasks considered cover both object classiﬁcation and non-classiﬁcation tasks in computer vision, such as depth estimation and 3D (occlusion) edge detection.

Original: 5. V ALIDATION OF TRANSFER PERFORMANCE
Revised: 5. V ALIDATION OF TRANSFER PERFORMANCE

Original: To demonstrate that our transferability metric is indeed a suitable measurement for task transfer performance, we compare it with the empirical performance of transfering features learned from ImageNet 1000 -class classiﬁcation (Krizhevsky et al.
Revised: To demonstrate that our transferability metric is indeed a suitable measurement for task transfer performance, we compare it with the empirical performance of transfering features learned from ImageNet 1000 -class classiﬁcation (Krizhevsky et al.

Original: (2012)) to Cifar 100 -class classiﬁcation (Krizhevsky &
Revised: (2012)) to Cifar 100 -class classiﬁcation (Krizhevsky &

Original: Hinton (2009)), using a network similar to Figure 1.
Revised: Hinton (2009)), using a network similar to Figure 1.

Original: Comparing to ImageNet-1000, Cifar-100 has smaller sample size and its images have lower resolution.
Revised: Comparing to ImageNet-1000, Cifar-100 has smaller sample size and its images have lower resolution.

Original: Therefore it is considered to be a more challenging task than ImageNet, making it a suitable case for transfer learning.
Revised: Therefore it is considered to be a more challenging task than ImageNet, making it a suitable case for transfer learning.

Original: In addition, we use a pretrained ResNet- 50 as the source model due to its high performance and regular structure.
Revised: In addition, we use a pretrained ResNet- 50 as the source model due to its high performance and regular structure.

Original: Validation of H-score.
Revised: Validation of H-score.

Original: The training data for the target task in this experiemnt consists of 20 ,images randomly sampled from Cifar- 100 .
Revised: The training data for the target task in this experiemnt consists of 20 ,images randomly sampled from Cifar- 100 .

Original: It is further split 9:1 into a training set and a testing set.
Revised: It is further split 9:1 into a training set and a testing set.

Original: The transfer network was trained using stochastic gradient descent with batch size 20 , 000 forepochs.
Revised: The transfer network was trained using stochastic gradient descent with batch size 20 , 000 forepochs.

Original: Figure 4.a compares the H-score of transferring from ﬁve different layers (4a-4f) in the source network with the target log-loss and test accuracy of the respective features.
Revised: Figure 4.a compares the H-score of transferring from ﬁve different layers (4a-4f) in the source network with the target log-loss and test accuracy of the respective features.

Original: As H-score increases, log-loss of the target network decreases almost linearly while the training and testing accuracy increase.
Revised: As H-score increases, log-loss of the target network decreases almost linearly while the training and testing accuracy increase.

Original: Such behavior is consistent with our expectation that H-score reﬂects the learning performance of the target task.
Revised: Such behavior is consistent with our expectation that H-score reﬂects the learning performance of the target task.

Original: We also demonstrated that target sample size does not affect the relationship between H-score and log-loss (Figure 4.b).
Revised: We also demonstrated that target sample size does not affect the relationship between H-score and log-loss (Figure 4.b).

Original: This experiment also showed another potential application of H-score for selecting the most suitable layer for ﬁne-tuning in transfer learning.
Revised: This experiment also showed another potential application of H-score for selecting the most suitable layer for ﬁne-tuning in transfer learning.

Original: In the example, transfer performance is better when an upper layer of the source networks is transferred.
Revised: In the example, transfer performance is better when an upper layer of the source networks is transferred.

Original: This could be because the target task and the source task are inherently similar such that the representation learned for one task can still be discriminative for the other.
Revised: This could be because the target task and the source task are inherently similar such that the representation learned for one task can still be discriminative for the other.

Original: [Figure]
Revised: [Figure]

Original: Validation of Transferability.
Revised: Validation of Transferability.

Original: We further tested our transferability metric for selecting the best target task for a given source task.
Revised: We further tested our transferability metric for selecting the best target task for a given source task.

Original: In particular, we constructed 4 target classiﬁcation tasks with 3, 5, 10, and 20 object categories from the Cifar-100 dataset.
Revised: In particular, we constructed 4 target classiﬁcation tasks with 3, 5, 10, and 20 object categories from the Cifar-100 dataset.

Original: We then computed the transferability from ImageNet-1000 (using the feature representation of layer 4f) to the target tasks.
Revised: We then computed the transferability from ImageNet-1000 (using the feature representation of layer 4f) to the target tasks.

Original: The results are compared to the empirical transfer performance trained with batch size 64 for 50 epochs in Figure4.c.
Revised: The results are compared to the empirical transfer performance trained with batch size 64 for 50 epochs in Figure4.c.

Original: We observe a similar behavior as the H-score in the case of a single target task in Figure 4.a, showing that transferability can directly predict the empirical transfer performance.
Revised: We observe a similar behavior as the H-score in the case of a single target task in Figure 4.a, showing that transferability can directly predict the empirical transfer performance.

Original: 5. T ASK TRANSFER FOR 3D SCENE UNDERSTANDING
Revised: 5. T ASK TRANSFER FOR 3D SCENE UNDERSTANDING

Original: In this experiment, we solve the source task selection problem for a collection of 3D sceneunderstanding tasks using the Taskonomy dataset from Zamir et al.
Revised: In this experiment, we solve the source task selection problem for a collection of 3D sceneunderstanding tasks using the Taskonomy dataset from Zamir et al.

Original: In the following, we will introduce the experiment setting and explain how we adapt the transferability metric to pixel-to-pixel recognition tasks.
Revised: In the following, we will introduce the experiment setting and explain how we adapt the transferability metric to pixel-to-pixel recognition tasks.

Original: Then we compare transferability with task afﬁnity , an empirical transferability metric proposed by Zamir et al.
Revised: Then we compare transferability with task afﬁnity , an empirical transferability metric proposed by Zamir et al.

Original: Data and Tasks.
Revised: Data and Tasks.

Original: The Taskonomy dataset contains 4,000,000 images of indoor scenes ofbuildings.
Revised: The Taskonomy dataset contains 4,000,000 images of indoor scenes ofbuildings.

Original: Every image has annotations for 26 computer vision tasks.
Revised: Every image has annotations for 26 computer vision tasks.

Original: We randomly sampled 20 ,images as training data.
Revised: We randomly sampled 20 ,images as training data.

Original: Eight tasks were chosen for this experiment, covering both classiﬁcations and lower-level pixel-to-pixel tasks.
Revised: Eight tasks were chosen for this experiment, covering both classiﬁcations and lower-level pixel-to-pixel tasks.

Original: Table 6 summaries the speciﬁcations of these tasks and sample outputs are shown in Figure 5.
Revised: Table 6 summaries the speciﬁcations of these tasks and sample outputs are shown in Figure 5.

Original: Feature Extraction and Data Preprocessing.
Revised: Feature Extraction and Data Preprocessing.

Original: For each task, Zamir et al.
Revised: For each task, Zamir et al.

Original: (2018) trained a fully supervised network with an encoder-decoder structure.
Revised: (2018) trained a fully supervised network with an encoder-decoder structure.

Original: When testing the transfer performance from
Revised: When testing the transfer performance from

Original: T S to T T , the encoder output of T S is used for training the decoder of T T .
Revised: T S to T T , the encoder output of T S is used for training the decoder of T T .

Original: For a fair comparison, we use the same trained encoders to extract source features.
Revised: For a fair comparison, we use the same trained encoders to extract source features.

Original: The output dimension of all encoders are 16 × 16 × 

Revised: The output dimension of all encoders are 16 × 16 ×

Original: and we ﬂatten the output into a vector of length 2048 .
Revised: and we ﬂatten the output into a vector of length 2048 .

Original: To reduce the computational complexity, we also resize the ground truth images into 64 × 64 .
Revised: To reduce the computational complexity, we also resize the ground truth images into 64 × 64 .

Original: For classiﬁcation tasks, H-score can be easily calculated given the source features.
Revised: For classiﬁcation tasks, H-score can be easily calculated given the source features.

Original: But for pixel-topixel tasks such as Edges and Depth, their ground truths are represented as images, which can not be quantized easily.
Revised: But for pixel-topixel tasks such as Edges and Depth, their ground truths are represented as images, which can not be quantized easily.

Original: As a workaround, we cluster the pixel values in the ground truth images into a palette of 16 colors.
Revised: As a workaround, we cluster the pixel values in the ground truth images into a palette of 16 colors.

Original: Then compute the H-score of the source features with respect to each pixel, before aggregating them into a ﬁnal score by averaging over the whole image.
Revised: Then compute the H-score of the source features with respect to each pixel, before aggregating them into a ﬁnal score by averaging over the whole image.

Original: We ran the experiment on a workstation with 3 .
Revised: We ran the experiment on a workstation with 3 .

Original: 40 GHz × 8 CPU and 16 GB memory.
Revised: 40 GHz × 8 CPU and 16 GB memory.

Original: Each pairwise
Revised: Each pairwise

Original: H-score computation ﬁnished in less than one hour including preprocessing.
Revised: H-score computation ﬁnished in less than one hour including preprocessing.

Original: Then we rank the source tasks according to their H-scores of a given target task and compare the ranking with that in Zamir et al.
Revised: Then we rank the source tasks according to their H-scores of a given target task and compare the ranking with that in Zamir et al.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Pairwise Transfer Results.
Revised: Pairwise Transfer Results.

Original: Source task ranking results using transferability and afﬁnity are visualized side by side in Figure 7, with columns representing source tasks and rows representing target tasks.
Revised: Source task ranking results using transferability and afﬁnity are visualized side by side in Figure 7, with columns representing source tasks and rows representing target tasks.

Original: For classiﬁcation tasks (the bottom two rows in the transferability matrix), the top two transferable source tasks are identical for both methods.
Revised: For classiﬁcation tasks (the bottom two rows in the transferability matrix), the top two transferable source tasks are identical for both methods.

Original: The best source task is the target task itself, as the encoder is trained on a task-speciﬁc network with much larger sample size.
Revised: The best source task is the target task itself, as the encoder is trained on a task-speciﬁc network with much larger sample size.

Original: Scene Class.
Revised: Scene Class.

Original: and
Revised: and

Original: Object Class.
Revised: Object Class.

Original: are ranked second for each other, as they are semantically related.
Revised: are ranked second for each other, as they are semantically related.

Original: Similar observations can be found in 2D pixel-to-pixel tasks (top two rows).
Revised: Similar observations can be found in 2D pixel-to-pixel tasks (top two rows).

Original: The results on lower rankings are noisier.
Revised: The results on lower rankings are noisier.

Original: A slightly larger difference between the two rankings can be found in 3D pixel-to-pixel tasks, especially 3D Occlusion Edges and 3D Keypoints.
Revised: A slightly larger difference between the two rankings can be found in 3D pixel-to-pixel tasks, especially 3D Occlusion Edges and 3D Keypoints.

Original: Though the top four ranked tasks of both methods are exactly the four 3D tasks.
Revised: Though the top four ranked tasks of both methods are exactly the four 3D tasks.

Original: It could indicate that these low level vision tasks are closely related to each other so that the transferability among them are inherently ambiguous.
Revised: It could indicate that these low level vision tasks are closely related to each other so that the transferability among them are inherently ambiguous.

Original: We also computed the ranking correlations between transferability and afﬁnity using Spearman’s R and Discounted
Revised: We also computed the ranking correlations between transferability and afﬁnity using Spearman’s R and Discounted

Original: Cumulative Gain (DCG).
Revised: Cumulative Gain (DCG).

Original: Both criterion show positive correlations for all target tasks.
Revised: Both criterion show positive correlations for all target tasks.

Original: The correlation is especially strong with DCG as higher ranking entities are given larger weights.
Revised: The correlation is especially strong with DCG as higher ranking entities are given larger weights.

Original: The above observations inspire us to deﬁne a notion of task relatedness, as some tasks are frequently ranked high for each other.
Revised: The above observations inspire us to deﬁne a notion of task relatedness, as some tasks are frequently ranked high for each other.

Original: Speciﬁcally, we represent each task with a vector consisting of H-scores of all the source tasks, then apply agglomerative clustering over the task vectors.
Revised: Speciﬁcally, we represent each task with a vector consisting of H-scores of all the source tasks, then apply agglomerative clustering over the task vectors.

Original: As shown in the dendrogram in Figure 7, 2D tasks and most 3D tasks are grouped into different clusters, but on a higher level, all pixel-to-pixel tasks are considered one category compared to the classiﬁcations tasks.
Revised: As shown in the dendrogram in Figure 7, 2D tasks and most 3D tasks are grouped into different clusters, but on a higher level, all pixel-to-pixel tasks are considered one category compared to the classiﬁcations tasks.

Original: Higher Order Transfer.
Revised: Higher Order Transfer.

Original: Sometimes we need to combine features from two or more source tasks for better transfer performance.
Revised: Sometimes we need to combine features from two or more source tasks for better transfer performance.

Revised: A common way to combine features from multiple models in deep neural networks is feature concatenation.

Original: Our transferability deﬁnition can be easily adapted to high order feature transfer, by concatenating features from multiple source tasks.
Revised: For such problems, our transferability deﬁnition can be easily adapted to high order feature transfer, by computing the H-score of the concatenated features.

Original: Figure 8 shows the ranking results of all combinations of source task pairs for each target task.
Revised: Figure 8 shows the ranking results of all combinations of source task pairs for each target task.

Original: For all tasks except for 3D Occlusion Edges and Depth, the best seond-order source feature is the combination of the top two tasks of the ﬁrst-order ranking.
Revised: For all tasks except for 3D Occlusion Edges and Depth, the best seond-order source feature is the combination of the top two tasks of the ﬁrst-order ranking.

Original: We examine the exception in Figure 9, by visualizing the pixel-by-pixel H-scores of ﬁrst and second order transfers to Depth using a heatmap (lighter color implies a higher H-score).
Revised: We examine the exception in Figure 9, by visualizing the pixel-by-pixel H-scores of ﬁrst and second order transfers to Depth using a heatmap (lighter color implies a higher H-score).

Original: Note that different source tasks can be good at predicting different parts of the image.
Revised: Note that different source tasks can be good at predicting different parts of the image.

Original: The top row shows the results of combining tasks with two different “transferability patterns” while the bottom row shows those with similar patterns.
Revised: The top row shows the results of combining tasks with two different “transferability patterns” while the bottom row shows those with similar patterns.

Original: Combining tasks with different transferability patterns has a more signiﬁcant improvement to the overall performance of the target task.
Revised: Combining tasks with different transferability patterns has a more signiﬁcant improvement to the overall performance of the target task.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Transfer learning.
Revised: Transfer learning.

Original: Transfer learning can be devided into two categories: domain adaptation , where knowledge transfer is achieved by making representations learned from one input domain work on a different input domain, e.g. adapt models for RGB images to infrared images (Wang & Deng (2018)); and task transfer learning , where knowledge is transferred between different tasks on the same input domain (Torrey & Shavlik (2010)).
Revised: Transfer learning can be devided into two categories: domain adaptation , where knowledge transfer is achieved by making representations learned from one input domain work on a different input domain, e.g. adapt models for RGB images to infrared images (Wang & Deng (2018)); and task transfer learning , where knowledge is transferred between different tasks on the same input domain (Torrey & Shavlik (2010)).

Original: Our paper focus on the latter prolem.
Revised: Our paper focus on the latter prolem.

Original: Empirical studies on transferability.
Revised: Empirical studies on transferability.

Original: Yosinski et al.
Revised: Yosinski et al.

Original: compared the transfer accuracy of features from different layers in a neural network between image classiﬁcation tasks.
Revised: compared the transfer accuracy of features from different layers in a neural network between image classiﬁcation tasks.

Original: A similar study was performed for NLP tasks by Conneau et al.
Revised: A similar study was performed for NLP tasks by Conneau et al.

Original: Zamir et al.
Revised: Zamir et al.

Original: (2018) determined the optimal transfer hierarchy over a collection of perceptual indoor scene understanidng tasks, while transferability was measured by a non-parameteric score called ”task afﬁnity” derived from neural network transfer losses coupled with an ordinal normalization scheme.
Revised: (2018) determined the optimal transfer hierarchy over a collection of perceptual indoor scene understanidng tasks, while transferability was measured by a non-parameteric score called ”task afﬁnity” derived from neural network transfer losses coupled with an ordinal normalization scheme.

Original: Task relatedness.
Revised: Task relatedness.

Original: One approach to deﬁne task relatedness is based on task generation.
Revised: One approach to deﬁne task relatedness is based on task generation.

Original: Generalization bounds have been derived for multi-task learning (Baxter (2000)), learning-to-learn (Maurer (2009)) and life-long learning (Pentina & Lampert (2014)).
Revised: Generalization bounds have been derived for multi-task learning (Baxter (2000)), learning-to-learn (Maurer (2009)) and life-long learning (Pentina & Lampert (2014)).

Original: Although these studies show theoretical results on transferability, it is hard to infer from data whether the assumptions are satisﬁed.
Revised: Although these studies show theoretical results on transferability, it is hard to infer from data whether the assumptions are satisﬁed.

Original: Another approach is estimating task relatedness from data, either explicitly (Bonilla et al.
Revised: Another approach is estimating task relatedness from data, either explicitly (Bonilla et al.

Original: ; Zhang (2013)) or implicitly as a regularization term on the network weights (Xue et al.
Revised: ; Zhang (2013)) or implicitly as a regularization term on the network weights (Xue et al.

Original: ; Jacob et al.
Revised: ; Jacob et al.

Original: Most works in this category are limited to shallow ones in terms of the model parameters.
Revised: Most works in this category are limited to shallow ones in terms of the model parameters.

Original: Representation learning and evaluation.
Revised: Representation learning and evaluation.

Original: Selecting optimal features for a given task is traditionally performed via feature subset selection or feature weight learning.
Revised: Selecting optimal features for a given task is traditionally performed via feature subset selection or feature weight learning.

Original: Subset selection chooses features with maximal relevance and minimal redundancy according to information theoretic or statistical criteria (Peng et al.
Revised: Subset selection chooses features with maximal relevance and minimal redundancy according to information theoretic or statistical criteria (Peng et al.

Original: ; Hall (1999)).
Revised: ; Hall (1999)).

Original: The feature weight approach learns the task while regularizing feature weights with sparsity constraints, which is common in multi-task learningLiao &
Revised: The feature weight approach learns the task while regularizing feature weights with sparsity constraints, which is common in multi-task learningLiao &

Original: Carin (2006); Argyriou et al.
Revised: Carin (2006); Argyriou et al.

Original: In a different perspective, Huang et al.
Revised: In a different perspective, Huang et al.

Original: (2017) consider the universal feature selection problem, which ﬁnds the most informative features from data when the exact inference problem is unknown.
Revised: (2017) consider the universal feature selection problem, which ﬁnds the most informative features from data when the exact inference problem is unknown.

Original: When the target task is given, the universal feature is equivalent to the minimum error probability feature used in this work.
Revised: When the target task is given, the universal feature is equivalent to the minimum error probability feature used in this work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented H-score, an information theoretic approach to estimating the performance of features when transferred across classiﬁcation tasks.
Revised: In this paper, we presented H-score, an information theoretic approach to estimating the performance of features when transferred across classiﬁcation tasks.

Original: Then we used it to deﬁne a notion of task transferability in multi-task transfer learning problems, that is both time and sample complexity efﬁcient.
Revised: Then we used it to deﬁne a notion of task transferability in multi-task transfer learning problems, that is both time and sample complexity efﬁcient.

Original: The resulting transferability metric also has a strong operational meaning as the ratio between the best achievable error exponent of the transferred representation and the minium error exponent of the target task.
Revised: The resulting transferability metric also has a strong operational meaning as the ratio between the best achievable error exponent of the transferred representation and the minium error exponent of the target task.

Original: Our transferability score successfully predicted the performance for transfering features from
Revised: Our transferability score successfully predicted the performance for transfering features from

Original: ImageNet-1000 classiﬁcation task to Cifar-100 task.
Revised: ImageNet-1000 classiﬁcation task to Cifar-100 task.

Original: Moreover, we showed how the transferability metric can be applied to a set of diverse computer vision tasks using the Taskonomy dataset.
Revised: Moreover, we showed how the transferability metric can be applied to a set of diverse computer vision tasks using the Taskonomy dataset.

Original: In future works, we plan to extend our theoretical results to non-classiﬁcation tasks, as well as relaxing the local assumptions on the conditional distributions of the tasks.
Revised: In future works, we plan to extend our theoretical results to non-classiﬁcation tasks, as well as relaxing the local assumptions on the conditional distributions of the tasks.

Revised: We will also investigate properties of higher order transferability, developing more scalable algorithms that avoid computing the H-score of all task pairs.

Original: On the application side, as transferability tells us how different tasks are related, we hope to use this information to design better task hierarchies for transfer learning.
Revised: On the application side, as transferability tells us how different tasks are related, we hope to use this information to design better task hierarchies for transfer learning

Original: A PPENDIX A I NFERENCE USING L OCAL I NFORMATION G EOMETRY 
Revised: A PPENDIX A I NFERENCE USING L OCAL I NFORMATION G EOMETRY A. A PRIMER ON ERROR EXPONENTS

Original: Consider the binary hypothesis testing problem over m samples { x ( i ) } mi =1 drawn i.i.d from distributions P 1 or P 2 , with P 1 , P 2 ∈ N (cid:15) ( P X 0 ) .
Revised: To begin with, consider the binary hypothesis testing problem over m i.i.d. sampled observations { x ( i ) } m i =1 (cid:44) x m with the following hypotheses: H 0 :

Revised: x m ∼ P 1 or H 1 : x m ∼ P 2 .

Original: Let φ i 
Revised: Let P x m be the empirical distribution of the samples. The optimal test, i e., the log likelihood ratio test can be stated in terms of information-theoretic quantities as follows: [Equation] Further, using Sannov’s theorem we have that asymptotically the probability of type I error

Revised: [Equation]

Revised: where P ∗ 1 = argmin P ∈

Revised: A c D ( P || P 1 ) and A c ( T ) =

Revised: { x m : D ( P x m || P 2 ) − D ( P x m || P 1 ) < 1 m log T } denotes the rejection region.

Revised: Similarly, for type II error

Revised: [Equation]

Revised: where P ∗ 2 = argmin P ∈

Revised: A D ( P || P 2 ) and A = { x m : D ( P x m || P 2 ) − D ( P x m || P 1 ) > 1 m log T } represents the acceptance region.

Revised: (See Figure 10)

Revised: The overall probability of error is P ( m ) e =

Original: = P i ( x ) 
Revised: αPr ( H 0 ) + βPr ( H 1 ) and the best achievable exponent in the Bayesian probability of error (a.k.a. Chernoff exponent) is deﬁned as: [Equation]

Original: [Equation]
Revised: See Cover & Thomas (1991) for more background information on error exponents and its related theorems. A. L OCAL INFORMATION GEOMETRY Now consider the same binary hypothesis testing problem, but with the local constraint P 1 , P 2 ∈

Original: [Equation]
Revised: N (cid:15) ( P X 0 ) . Let φ i

Original:  x [Equation] x ) denote the information vectors corresponding to P i for i = 1 , 2 .
Revised: = P i ( x ) − P X 0 ( x ) (cid:15) √ P X 0 ( x ) denote the information vectors corresponding to P i for i = 1 , 2 .

Original: Makur et al.
Revised: Makur et al.

Original: (2015) uses local information geometry to connect the error exponent in hypothesis testing to the length of certain information vectors, summarized in the following two lemmas.
Revised: (2015) uses local information geometry to connect the error exponent in hypothesis testing to the length of certain information vectors, summarized in the following two lemmas.

Original: Lemma 1.
Revised: Lemma 1.

Original: Given zero-mean, unit variance feature function f ( x ) : X → R , the optimal error exponent (a.k.a.
Revised: Given zero-mean, unit variance feature function f ( x ) : X → R , the optimal error exponent (a.k.a.

Original: Chernoff exponent) of this hypothesis testing problem is
Revised: Chernoff exponent) of this hypothesis testing problem is

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Given zero-mean, unit variance feature function f ( x ) : X → R , the error exponent of a mismatched decision function of the form l = 1 m (cid:80) mi =1 ( f ( x ( i ) )) is
Revised: Given zero-mean, unit variance feature function f ( x ) : X → R , the error exponent of a mismatched decision function of the form l = 1 m (cid:80) mi =1 ( f ( x ( i ) )) is

Original: [Equation]
Revised: [Equation]

Original: where ξ ( x ) = (cid:112) P 0 ( x ) f ( x ) is the feature vector associated with f ( x ) .
Revised: where ξ ( x ) = (cid:112) P 0 ( x ) f ( x ) is the feature vector associated with f ( x ) .

Original: As our discussion of transferability mostly concerns with multi-dimensional features, we present the k -dimensional generalization of Lemma 2 below: (Equation 1 in the main paper.)
Revised: As our discussion of transferability mostly concerns with multi-dimensional features, we present the k -dimensional generalization of Lemma 2 below: (Equation 1 in the main paper.)

Original: Lemma 3.
Revised: Lemma 3.

Original: Given k normalized feature functions f ( x ) =
Revised: Given k normalized feature functions f ( x ) =

Original: [ f 1 ( x ) , . .
Revised: [ f 1 ( x ) , . .

Original: , f k ( x )] , such that E [ f i ( X )]
Revised: , f k ( x )] , such that E [ f i ( X )]

Original: =for all i , and cov ( f ( X ))
Revised: =for all i , and cov ( f ( X ))

Original: =
Revised: =

Original: I , we deﬁne a k -d statistics of the form l k =
Revised: I , we deﬁne a k -d statistics of the form l k =

Original: ( l 1 , ..., l k ) where l i = 

Revised: ( l 1 , ..., l k ) where l i =

Original: m (cid:80) m l =1 f i ( x ( l ) ) .
Revised: m (cid:80) ml =1 f i ( x ( l ) ) .

Original: Let Ξ( x ) =
Revised: Let Ξ( x ) =

Original: [ ξ 1 ( x ) , . . . , ξ k ( x )] be the corresponding feature vectors with ξ i ( x ) =
Revised: [ ξ 1 ( x ) , . . . , ξ k ( x )] be the corresponding feature vectors with ξ i ( x ) =

Original: (cid:112) P X ( x ) f i ( x ) .
Revised: (cid:112) P X ( x ) f i ( x ) .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Cram´er’s theorem, the error exponent under P i is
Revised: According to Cram ´ er’s theorem, the error exponent under P i is

Original: [Equation]
Revised: [Equation]

Original: where δ ( λ ) (cid:44) (cid:8) P : E P [ f ( X ) k ] =
Revised: where δ ( λ ) (cid:44) (cid:8) P : E P [ f ( X ) k ] =

Original: λ E P 1 [ f ( X ) k ] + (1 − λ ) E P 2 [ f ( X ) k ] (cid:9) .
Revised: λ E P 1 [ f ( X ) k ] + (1 − λ ) E P 2 [ f ( X ) k ] (cid:9) .

Original: With the techniques developed in local information geometry, the above above problem is equivalent to the following problem:
Revised: With the techniques developed in local information geometry, the above above problem is equivalent to the following problem:

Original: [Equation]
Revised: [Equation]

Original: where σ ( λ ) (cid:44) (cid:110) ˜
Revised: where σ ( λ ) (cid:44) (cid:110) ˜

Original: φ θ k : (cid:68) ˜
Revised: φ θ k : (cid:68) ˜

Original: φ θ k , ξ l (cid:69) =
Revised: φ θ k , ξ l (cid:69) =

Original: (cid:104) λφ 1 + (1 − λ )
Revised: (cid:104) λφ 1 + (1 − λ )

Original: φ 2 , ξ l (cid:105) , l = 1 , ..., k (cid:111) and i = 1 , 2 .
Revised: φ 2 , ξ l (cid:105) , l = 1 , ..., k (cid:111) and i = 1 , 2 .

Original: Further, using the equation ˜ φ θ k ( x ) = (cid:80) kl
Revised: Further, using the equation ˜ φ θ k ( x ) = (cid:80) kl

Original: =

Revised: =

Original: θ l ξ l ( x )+ φ i ( x ) − α ( θ k ) (cid:112) P 0 ( x )+ o ( (cid:15) ) and the equation α ( θ k )
Revised: θ l ξ l ( x )+ φ i ( x ) − α ( θ k ) (cid:112) P 0 ( x )+ o ( (cid:15) ) and the equation α ( θ k )

Original: =
Revised: =

Original: o ( (cid:15) 2 )
Revised: o ( (cid:15) 2 )

Original: , it is easy to show that E 1 ( λ ) =
Revised: , it is easy to show that E 1 ( λ ) =

Original: E 1 ( λ ) when λ
Revised: E 1 ( λ ) when λ

Original: = 12 .
Revised: = 12 .

Original: Then the overall error probability has the exponent as shown in Equation (1). 
Revised: Then the overall error probability has the exponent as shown in Equation (1). [Equation]

Original: A PPENDIX B P ROOF OF T HEOREM
Revised: C ORRELATION AND T HE ACE A LGORITHM

Revised: Given random variables X and Y , the HGR maximal correlation ρ ( X ; Y ) deﬁned in Equation 2 is a generalization of the Pearson’s correlation coefﬁcient to capture non-linear dependence between random variables.

Revised: According to R´enyi (1959), it satisﬁes all seven natural postulates of a suitable dependence measure.

Revised: Some notable properties are listed below:

Revised: ρ ( X ; Y ) is deﬁned for any pair of random variables X

Original: {+and Y 2. 
+}
Revised: {+and Y 2. 
+}

Revised: ≤ ρ ( X ; Y ) ≤ 1 ; 3.

Original: {+ρ ( X ; Y ) = 
+}
Revised: {+ρ ( X ; Y ) = 
+}

Revised: if and only if X and Y are independent 4.

Revised: ρ ( X ; Y )

Original: {+= 
+}
Revised: {+= 
+}

Revised: if and only if X and Y are strictly dependent.

Revised: When the feature dimension is 1, the solution of the maximal HGR correlation is ρ ( X ; Y ) =

Revised: σ 1 , the largest singular value of the DTM matrix ˜ B .

Revised: For k -dimensional features, ρ ( X ; Y ) =

Revised: (cid:80) ki ( σ i ) .

Revised: However, computing ˜ B requires estimating the joint probability P Y X from data, which is inpractical in real applications.

Revised: Breiman & Friedman (1985) proposed an efﬁcient algorithm, alternating condition expectation (ACE), inspired by the power method for computing matrix singular values.

Revised: Require training

Revised: , m }

Revised: In Algorithm 1, we ﬁrst initialize g as a random k -dimensional zero-mean function.

Revised: Then iteratively update f ( x ) and g ( y ) for all x ∈ X and y ∈ Y .

Revised: The conditional distribution on Line 4 and 6 are computed as the empirical average over m samples.

Revised: The normalization steps on Lines 5 and 7 can also be implemented using the Gram-Schmidt process.

Revised: Note that the ACE algorithm has several variations in previous works, including a kernel-smoothed version (Breiman & Friedman (1985)) and a parallel version with improved efﬁciency (Huang et al.

Revised: An alternative formulation that supports continuous X and large feature dimension k has also been proposed recently (Wang et al.

Revised: Next we look at the convergence property of the ACE algorithm.

Revised: Let f ( X ) , g ( Y ) be the true maximal correlation functions, and let ˜ f

Revised: ( X ) , ˜ g

Revised: ( Y ) be estimations computed with Algorithm 1 from m i.i.d.

Revised: sampled training data.

Revised: Similarly, denote by E [ f ( X ) g

Revised: ( Y )] and ˆ E m [ ˜ f ( X )˜

Revised: g

Revised: ( Y )] the true and estimated maximal correlations, respectively.

Revised: Using Sanov’s Theorem, we can show that for a small ∆ > 0 , the probability that the ratio between the true and estimated maximal correlation is within 1 ± ∆ drops exponentially as the number of samples increases.

Revised: Hence the ACE algorithm converges in exponential time.

Revised: The following theorem gives the precise sampling complexity for k = 1 .

Revised: Theorem 2.

Revised: For any random variables X and Y with joint probability distribution P Y X , if X and Y are not independent, then for any f : X → R and g :

Revised: Y → R such that E [ f 2 ( X )]

Revised: =

Revised: E [ g 2 ( Y )]

Revised: = 1 , we have that

Revised: [Equation]

Revised: for any given ∆ > 0 .

Revised: [Equation]

Revised: PPENDIX C P ROOF OF T HEOREM

Original: To simplify the proof, we ﬁrst consider the case when the feature function is 1-dimensional.i.e.
Revised: To simplify the proof, we ﬁrst consider the case when the feature function is 1-dimensional.i.e.

Original: f : X → R .
Revised: f : X → R .

Original: We have the following lemma:
Revised: We have the following lemma:

Original: Lemma 4.
Revised: Lemma 4.

Original: Let f : X → R be an arbitrary feature function of X such that E
Revised: Let f : X → R be an arbitrary feature function of X such that E

Original: [ f ( X )]
Revised: [ f ( X )]

Original: = 0 and
Revised: = 0 and

Original: E [ f ( X ) 2 ] = 1 , then || ˜
Revised: E [ f ( X ) 2 ] = 1 , then || ˜

Original: Bξ || 2 F = E P Y
Revised: Bξ || 2 F = E P Y

Original: [( E [ f ( X ) | Y ]) 2 ] where ξ is the feature vector corresponding to f .
Revised: [( E [ f ( X ) | Y ]) 2 ] where ξ is the feature vector corresponding to f .

Original: Proof.
Revised: Proof.

Original: Since ξ ( x ) = (cid:112) P X ( x ) f ( x ) , we have
Revised: Since ξ ( x ) = (cid:112) P X ( x ) f ( x ) , we have

Original: [Equation]
Revised: [Equation]

Original: The last equality uses the assumption that E [ f ( x )] = 0 .
Revised: The last equality uses the assumption that E [ f ( x )] = 0 .

Original: Theorem 2 (1D version of Theorem 1) .
Revised: Theorem 3 (1D version of Theorem 1) .

Original: Given P X | Y =0 , P X | Y =1 ∈ N X (cid:15) ( P 0 ,X ) and features f :
Revised: Given P X | Y =0 , P X | Y =1 ∈ N X (cid:15) ( P 0 ,X ) and features f :

Original: X → R such that E [ f ( X )]
Revised: X → R such that E [ f ( X )]

Original: = 0 and E [ f ( X ) 2 ] = 1 , then there exists some constant c independent of f such that
Revised: = 0 and E [ f ( X ) 2 ] = 1 , then there exists some constant c independent of f such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From E [ f ( X )] =
Revised: From E [ f ( X )] =

Original: P Y (0) E [ f ( X ) | Y = 0]
Revised: P Y (0) E [ f ( X ) | Y = 0]

Original: [Equation] P 
Revised: + P Y (1) E

Original: [Equation] 1) E [ f ( X ) | Y = 1] = 0 , we ﬁrst derive the following properties of the conditional expectations of f ( x ) :
Revised: [ f ( X ) | Y = 1] = 0 , we ﬁrst derive the following properties of the conditional expectations of f ( x ) :

Original: [Equation]
Revised: [Equation]

Original: On the R.H.S. of Equation 4, we apply Lemma 4 to write
Revised: On the R.H.S. of Equation 5, we apply Lemma 4 to write

Original: [Equation]
Revised: [Equation]

Original: Next consider the L [Equation] H.S. of the equation, by Lemma 2, we have E ( h ) =
Revised: Next consider the L.H.S. of the equation, by Lemma 2, we have E ( h ) =

Original: (cid:15) 2 8 (cid:104) ξ, φ 1 − φ 2 (cid:105) 

Revised: (cid:15) 2 8 (cid:104) ξ, φ 1 − φ 2 (cid:105)

Original: + o ( (cid:15) 2 ) =
Revised: + o ( (cid:15) 2 ) =

Original: c 0 (cid:104) ξ, φ 1 − φ 2 (cid:105) 2 for some constant c 0 .
Revised: c 0 (cid:104) ξ, φ 1 − φ 2 (cid:105) 2 for some constant c 0 .

Original: [Equation]
Revised: [Equation]

Original: )=
Revised: ) · ( E [ f ( X ) | Y = 1])( E [ f ( X ) | Y = 0]) = c(cid:18) P Y (0) + P Y (1) P Y (0) (cid:19)· ( E [ f ( X ) | Y = 1])= c 0 P Y (0) + P Y (1) P Y (0) P Y (1) (cid:18) P Y (0) P Y (1) + P Y (1)

Original:  c || ˜
Revised: P Y (0) (cid:19) · ( E [ f ( X ) | Y = 1])= c || ˜

Original: Bξ || 2 F
Revised: Bξ || 2 F

Original: For k ≥ 2 , Lemma 4 can be restated as follows:
Revised: For k ≥ 2 , Lemma 4 can be restated as follows:

Original: Lemma 5.
Revised: Lemma 5.

Original: Let f : X → R k be a k -dimensional feature function of X , where f ( x ) =
Revised: Let f : X → R k be a k -dimensional feature function of X , where f ( x ) =

Original: [ f 1 ( x ) , . .
Revised: [ f 1 ( x ) , . .

Original: , f k ( x )] .
Revised: , f k ( x )] .

Original: Further we assume that E [ f ( X )]
Revised: Further we assume that E [ f ( X )]

Original: = 0 and cov ( f ( X ))
Revised: = 0 and cov ( f ( X ))

Original: =
Revised: =

Original: E [ f ( X ) T f ( X )]
Revised: E [ f ( X ) T f ( X )]

Original: = I .
Revised: = I .

Original: Then || ˆ B Ξ || 2 F = tr ( cov ( E [ f ( X ) | Y ])) where columns of Ξ are the information vectors corresponding to f 1 ( x ) , . . .
Revised: Then || ˆ B Ξ || 2 F = tr ( cov ( E [ f ( X ) | Y ])) where columns of Ξ are the information vectors corresponding to f 1 ( x ) , . . .

Original: , f k ( x ) .
Revised: , f k ( x ) .

Original: Proof.
Revised: Proof.

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: where 1 is a column vector with all entries 1 and length |Y| .
Revised: where 1 is a column vector with all entries 1 and length |Y| .

Original: Since E [ f ( X )]
Revised: Since E [ f ( X )]

Original: = 0 , we have
Revised: = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Finally, we derive the multi-dimensional case for Theorem 1.
Revised: Finally, we derive the multi-dimensional case for Theorem 1.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Using Lemma 5 and a similar argument as in the simpliﬁed proof, the R.H.S of the equation becomes
Revised: Using Lemma 5 and a similar argument as in the simpliﬁed proof, the R.H.S of the equation becomes

Original: [Equation]
Revised: [Equation]

Original: By Lemma
Revised: By Lemma

Original: [Equation]
Revised: [Equation]

Original: the L
Revised: the L

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: S. of the equation can be written as E kf =
Revised: S. of the equation can be written as E kf =

Original: c [Equation] ki= [Equation]
Revised: c 0 (cid:80) ki

Original: φ

Original: [Equation] − φ [Equation] 2 for some constant c 0 .
Revised: = l (cid:104) ξ i , φ 1 − φ 2 (cid:105) 2 for some constant c 0 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Revised: ) | Y = 1]) T ( E [ f ( X ) | Y = 0] − E [ f ( X ) | Y = 1])

Revised: =

Revised: c 0 P Y (0)

Revised: + P Y (1)

Revised: P Y (0) P Y (1) (cid:18) P Y (0) P Y (1) + P Y (1)

Revised: P Y (0) (cid:19) · E [ f ( X ) | Y = 1] T E [ f ( X ) | Y = 1]

Revised: =

Revised: c (cid:107) ˜ B Ξ (cid:107) 2 F

Original: A PPENDIX C G ENERALIZATION OF H-S CORE FOR A RBITRARY F EATURE S ET
Revised: A PPENDIX D G ENERALIZATION OF H-S CORE FOR A RBITRARY F EATURE S ET

Original: Since Ξ = (cid:2) √ P X (cid:3) f( X ) and E [ f ( X )]
Revised: Since Ξ = (cid:2) √ P X (cid:3) f( X ) and E [ f ( X )]

Original: = 0 , we have
Revised: = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Equation (5) gives a more understandable expression of the normalization term.
Revised: Equation (6) gives a more understandable expression of the normalization term.

Original: We can also write ˜
Revised: We can also write ˜

Original: B Ξ as follows:
Revised: B Ξ as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 is a column vector with all entries 1 and length |Y|
Revised: where 1 is a column vector with all entries 1 and length |Y|

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: The last equality is derived by substituting(5) and (6)  into (7).
Revised: The last equality is derived by substituting (6) and (7) into (8).

Original: A PPENDIX D H- SCORE AND S OFTMAX R EGRESSION
Revised: A PPENDIX E H- SCORE AND S OFTMAX R EGRESSION

Original: In softmax regression, given m training examples { ( x ( i ) , y ( i ) ) } mi =1 , the cross-entropy loss of the model is
Revised: In softmax regression, given m training examples { ( x ( i ) , y ( i ) ) } mi =1 , the cross-entropy loss of the model is

Original: [Equation]
Revised: [Equation]

Original: Minimizing (cid:96) is equivalent to minimizing D (cid:0) P Y X
Revised: Minimizing (cid:96) is equivalent to minimizing D (cid:0) P Y X

Original: || P X Q Y | X (cid:1) where P Y X is the joint empirical distribution of ( X, Y ) .
Revised: || P X Q Y | X (cid:1) where P Y X is the joint empirical distribution of ( X, Y ) .

Original: Using information geometry, it can be shown that under a local assumption
Revised: Using information geometry, it can be shown that under a local assumption

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) which reveals a close connection between log loss and the modal decomposition of ˜ B .
Revised: 12) which reveals a close connection between log loss and the modal decomposition of ˜ B .

Original: In consequence, it is reasonable to measure the classiﬁcation performance with (cid:107) ˜ B − ΨΦ T (cid:107) 2 F given a pair of ( f, θ ) associated with (Ψ , Φ) .
Revised: In consequence, it is reasonable to measure the classiﬁcation performance with (cid:107) ˜ B − ΨΦ T (cid:107) 2 F given a pair of ( f, θ ) associated with (Ψ , Φ) .

Original: In the context of estimating transferability, we are interested in a one-sided problem, where Φ S is given by the source feature and Ψ becomes the only variable.[Equation]
Revised: In the context of estimating transferability, we are interested in a one-sided problem, where Φ S is given by the source feature and Ψ becomes the only variable.

Original: [Equation]
Revised: [Equation]

Original: Training the network is equivalent to ﬁnding the optimal weight Ψ ∗ that minimizes the log-loss.
Revised: Training the network is equivalent to ﬁnding the optimal weight Ψ ∗ that minimizes the log-loss.

Original: By taking the derivative of the Objective function with respect to Ψ , we get
Revised: By taking the derivative of the Objective function with respect to Ψ , we get

Original: [Equation]
Revised: [Equation]

Original: Substituting (13) in the Objective of (12), we can derive the following close-form solution for the log loss.
Revised: Substituting (14) in the Objective of (13), we can derive the following close-form solution for the log loss.

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term in (14) is ﬁxed given T T while the second term has exactly the form of H-score.
Revised: The ﬁrst term in (15) is ﬁxed given T T while the second term has exactly the form of H-score.

Original: This implies that log loss is negatively linearly related to H-score.
Revised: This implies that log loss is negatively linearly related to H-score.

Original: We demonstrates this relationship experimentally, using a collection of synthesized tasks (Figure 10).
Revised: We demonstrates this relationship experimentally, using a collection of synthesized tasks (Figure 11).

Original: In particular, the target task is generated based on a random stochastic matrix P Y 0 | X , and 20 source tasks are generated with the conditional probability matrix P Y i |
Revised: In particular, the target task is generated based on a random stochastic matrix P Y 0 | X , and 20 source tasks are generated with the conditional probability matrix P Y i |

Original: X =
Revised: X =

Original: P Y 

Revised: P Y

Original: | X + iλI for some positive constant λ .
Revised: | X + iλI for some positive constant λ .

Original: The universal minimum error probability features for each source task are used as the source features f S i ( x ) , while the respective log-loss are obtained through training a simple neural network in Figurewith cross-entropy loss.
Revised: The universal minimum error probability features for each source task are used as the source features f S i ( x ) , while the respective log-loss are obtained through training a simple neural network in Figurewith cross-entropy loss.

Original: The relationship is clearly linear with a constant offset.
Revised: The relationship is clearly linear with a constant offset.

Original: A PPENDIX E DTM AND M UTUAL I NFORMATION
Revised: A PPENDIX F DTM AND M UTUAL I NFORMATION

Original: Proposition 1.
Revised: Proposition 1.

Original: Under the local assumption that P X | Y ∈ N (cid:15) ( P X ) , mutual information I ( X ; Y ) = 12 || ˜ B
Revised: Under the local assumption that P X | Y ∈ N (cid:15) ( P X ) , mutual information I ( X ; Y ) = 12 || ˜ B

Original: || 2 F + o ( (cid:15) 2 ) , where ˜ B of the DTM matrix of X and Y .
Revised: || 2 F + o ( (cid:15) 2 ) , where ˜ B of the DTM matrix of X and Y .

Original: Proof.
Revised: Proof.

Original: First, we deﬁne φ X | Y y ( x ) =
Revised: First, we deﬁne φ X | Y y ( x ) =

Original: P X | Y ( x
Revised: P X | Y ( x

Original: | y ) − P X ( x ) (cid:15) √ P
Revised: | y ) − P X ( x ) (cid:15) √ P

Original: X ( x ) and let Φ X | Y ∈ R |X|×|Y| denote its matrix version.
Revised: X ( x ) and let Φ X | Y ∈ R |X|×|Y| denote its matrix version.

Original: Then we have 
Revised: Then we have (cid:15)

Original: [Equation]
Revised: [Equation]

Original: Next, we express the mutual information in terms of information vector φ X | Y ,
Revised: Next [Equation] we express the mutual information in terms of information vector φ X | Y ,

Original: [Equation]
Revised: [Equation]

Original: A PPENDIX F S UPPLEMENTARY R ESULTS ON E XPERIMENT 5. F
Revised: A PPENDIX G S UPPLEMENTARY R ESULTS ON E XPERIMENT 5. G. C OMPARISON OF H- SCORES AND A FFINITIES

Original: C OMPARISON OF H- SCORES AND A FFINITIES

Original: Table 1, with columns representing source tasks and rows representing target tasks.
Revised: Table 1, with columns representing source tasks and rows representing target tasks.

Original: For each target task, the upper row shows our results while the lower one shows the results in Zamir et al.
Revised: For each target task, the upper row shows our results while the lower one shows the results in Zamir et al.

Original: Score values are included in parenteses.
Revised: Score values are included in parenteses.

Original: [Table]
Revised: [Table]

Original: Here we present some detailed results on the comparison between H-score and the afﬁnity score in
Revised: Here we present some detailed results on the comparison between H-score and the afﬁnity score in

Original: Zamir et al.
Revised: Zamir et al.

Original: (2018) for pairwise transfer.
Revised: (2018) for pairwise transfer.

Original: The results of the classiﬁcation tasks are shown in Figure 11 and the results of Depth is shown in 12.
Revised: The results of the classiﬁcation tasks are shown in Figure 12 and the results of Depth is shown in 13.

Original: We can see in general, although afﬁnity and transferability have totally different value ranges, they tend to agree on the top few ranked tasks.
Revised: We can see in general, although afﬁnity and transferability have totally different value ranges, they tend to agree on the top few ranked tasks.

Original: F. Q UANTIZATION
Revised: G. L ABEL Q UANTIZATION

Original: During the quantization process of the pixel-to-pixel task labels (ground truth images), we are primarily concerned with two factors: computational complexity and information loss.
Revised: During the quantization process of the pixel-to-pixel task labels (ground truth images), we are primarily concerned with two factors: computational complexity and information loss.

Original: Too much information loss will lead to bad approximation of the original problems.
Revised: Too much information loss will lead to bad approximation of the original problems.

Original: On the other hand, having little information loss requires larger cluster size and computation cost.
Revised: On the other hand, having little information loss requires larger cluster size and computation cost.

Original: Figure (13) shows that even after quantization, much of the information in the images are retained.
Revised: Figure (14) shows that even after quantization, much of the information in the images are retained.

Original: To test the sensitivity of the cluster size, we used cluster centroids to recover the ground truth image pixel-by-pixel.
Revised: To test the sensitivity of the cluster size, we used cluster centroids to recover the ground truth image pixel-by-pixel.

Original: The 3D occlusion Edge detection results on a sample image is shown in Figure 14.
Revised: The 3D occlusion Edge detection results on a sample image is shown in Figure 15. are lost found

Revised: N = 16 strikes a good balance between recoverability and computation cost.