  For task-oriented dialog systems to be maximally useful, it must be able to
process conversations in a way that is (1) generalizable with a small number of
training examples for new task domains, and (2) robust to user input in various
styles, modalities or domains. In pursuit of these goals, we introduce the
RADDLE benchmark, a collection of corpora and tools for evaluating the
performance of models across a diverse set of domains. By including tasks with
limited training data, RADDLE is designed to favor and encourage models with a
strong generalization ability. RADDLE also includes a diagnostic checklist that
facilitates detailed robustness analysis in aspects such as language
variations, speech errors, unseen entities, and out-of-domain utterances. We
evaluate recent state-of-the-art systems based on pre-training and fine-tuning,
and find that grounded pre-training on heterogeneous dialog corpora performs
better than training a separate model per domain. Overall, existing models are
less than satisfactory in robustness evaluation, which suggests opportunities
for future improvement.
