Paper:
ABSTRACT An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems. Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature. This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies. Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classification, such as recognition tasks for 3D indoor-scene understanding.

Review:
The authors propose an information theoretic metric to determine a priori if a representation learned from a source task can be useful when learning another task. This is a timely topic and being able to compute such as metric between tasks upfront would be of great interest.
While the motivation and contributions of the paper are clearly stated, a number of questions remain. Central concepts like error exponent or maximal HGR correlation are mentioned in passing. The former is important to understand whether (1) makes sense and the latter is a fundamental quantity for this work. The surprising fact is that HGR is not discussed at all, nor is there any intuition provided of what this quantity represents.
While the proposed H-score is attractive when a source task has to be selected, as the denominator of the task transferability itself does not need to be computed, it is not straightforward to compute this quantity in general. The authors suggest to use the alternative conditional expectation (ACE) algorithm in order to optimize (2). ACE is not discussed in any detail (e.g., how accurate is this procedure; what are the choices and/or trade-offs if any?) and (2) is not justified. Overall, I felt section 4.2 was relatively inaccessible, but important as it indicates whether the proposed metric is of any use in practice.
Finally, I did not understand the argument that says the H-score is to be prepared over the mutual information. To me the mutual information is still the golden standard. It was also not clear why the proposed approach would not straightforwardly apply to non classification problems as suggested in the future work.