  The benefit of multi-task learning over single-task learning relies on the
ability to use relations across tasks to improve performance on any single
task. While sharing representations is an important mechanism to share
information across tasks, its success depends on how well the structure
underlying the tasks is captured. In some real-world situations, we have access
to metadata, or additional information about a task, that may not provide any
new insight in the context of a single task setup alone but inform relations
across multiple tasks. While this metadata can be useful for improving
multi-task learning performance, effectively incorporating it can be an
additional challenge. We posit that an efficient approach to knowledge transfer
is through the use of multiple context-dependent, composable representations
shared across a family of tasks. In this framework, metadata can help to learn
interpretable representations and provide the context to inform which
representations to compose and how to compose them. We use the proposed
approach to obtain state-of-the-art results in Meta-World, a challenging
multi-task benchmark consisting of 50 distinct robotic manipulation tasks.
