  Heterogeneous multi-task learning (HMTL) is an important topic in multi-task
learning (MTL). Most existing HMTL methods usually solve either scenario where
all tasks reside in the same input (feature) space yet unnecessarily the
consistent output (label) space or scenario where their input (feature) spaces
are heterogeneous while the output (label) space is consistent. However, to the
best of our knowledge, there is limited study on twofold heterogeneous MTL
(THMTL) scenario where the input and the output spaces are both inconsistent or
heterogeneous. In order to handle this complicated scenario, in this paper, we
design a simple and effective multi-task adaptive learning (MTAL) network to
learn multiple tasks in such THMTL setting. Specifically, we explore and
utilize the inherent relationship between tasks for knowledge sharing from
similar convolution kernels in individual layers of the MTAL network. Then in
order to realize the sharing, we weightedly aggregate any pair of convolutional
kernels with their similarity greater than some threshold $\rho$, consequently,
our model effectively performs cross-task learning while suppresses the
intra-redundancy of the entire network. Finally, we conduct end-to-end
training. Our experimental results demonstrate the effectiveness of our method
in comparison with the state-of-the-art counterparts.
