  Existing goal-oriented dialogue datasets focus mainly on identifying slots
and values. However, customer support interactions in reality often involve
agents following multi-step procedures derived from explicitly-defined company
policies as well. To study customer service dialogue systems in more realistic
settings, we introduce the Action-Based Conversations Dataset (ABCD), a
fully-labeled dataset with over 10K human-to-human dialogues containing 55
distinct user intents requiring unique sequences of actions constrained by
policies to achieve task success. We propose two additional dialog tasks,
Action State Tracking and Cascading Dialogue Success, and establish a series of
baselines involving large-scale, pre-trained language models on this dataset.
Empirical results demonstrate that while more sophisticated networks outperform
simpler models, a considerable gap (50.8% absolute accuracy) still exists to
reach human-level performance on ABCD.
