  Babies learn with very little supervision by observing the surrounding world.
They synchronize the feedback from all their senses and learn to maintain
consistency and stability among their internal states. Such observations
inspired recent works in multi-task and multi-modal learning, but existing
methods rely on expensive manual supervision. In contrast, our proposed
multi-task graph, with consensus shift learning, relies only on pseudo-labels
provided by expert models. In our graph, every node represents a task, and
every edge learns to transform one input node into another. Once initialized,
the graph learns by itself on virtually any novel target domain. An adaptive
selection mechanism finds consensus among multiple paths reaching a given node
and establishes the pseudo-ground truth at that node. Such pseudo-labels, given
by ensemble pathways in the graph, are used during the next learning iteration
when single edges distill this distributed knowledge. We validate our key
contributions experimentally and demonstrate strong performance on the Replica
dataset, superior to the very few published methods on multi-task learning with
minimal supervision.
