  The timeline of computer vision research is marked with advances in learning
and utilizing efficient contextual representations. Most of them, however, are
targeted at improving model performance on a single downstream task. We
consider a multi-task environment for dense prediction tasks, represented by a
common backbone and independent task-specific heads. Our goal is to find the
most efficient way to refine each task prediction by capturing cross-task
contexts dependent on tasks' relations. We explore various attention-based
contexts, such as global and local, in the multi-task setting and analyze their
behavior when applied to refine each task independently. Empirical findings
confirm that different source-target task pairs benefit from different context
types. To automate the selection process, we propose an Adaptive
Task-Relational Context (ATRC) module, which samples the pool of all available
contexts for each task pair using neural architecture search and outputs the
optimal configuration for deployment. Our method achieves state-of-the-art
performance on two important multi-task benchmarks, namely NYUD-v2 and
PASCAL-Context. The proposed ATRC has a low computational toll and can be used
as a drop-in refinement module for any supervised multi-task architecture.
