  We present a method for making neural network predictions robust to shifts
from the training data distribution. The proposed method is based on making
predictions via a diverse set of cues (called 'middle domains') and ensembling
them into one strong prediction. The premise of the idea is that predictions
made via different cues respond differently to a distribution shift, hence one
should be able to merge them into one robust final prediction. We perform the
merging in a straightforward but principled manner based on the uncertainty
associated with each prediction. The evaluations are performed using multiple
tasks and datasets (Taskonomy, Replica, ImageNet, CIFAR) under a wide range of
adversarial and non-adversarial distribution shifts which demonstrate the
proposed method is considerably more robust than its standard learning
counterpart, conventional deep ensembles, and several other baselines.
