  Dialogue policy learning based on reinforcement learning is difficult to be
applied to real users to train dialogue agents from scratch because of the high
cost. User simulators, which choose random user goals for the dialogue agent to
train on, have been considered as an affordable substitute for real users.
However, this random sampling method ignores the law of human learning, making
the learned dialogue policy inefficient and unstable. We propose a novel
framework, Automatic Curriculum Learning-based Deep Q-Network (ACL-DQN), which
replaces the traditional random sampling method with a teacher policy model to
realize the dialogue policy for automatic curriculum learning. The teacher
model arranges a meaningful ordered curriculum and automatically adjusts it by
monitoring the learning progress of the dialogue agent and the over-repetition
penalty without any requirement of prior knowledge. The learning progress of
the dialogue agent reflects the relationship between the dialogue agent's
ability and the sampled goals' difficulty for sample efficiency. The
over-repetition penalty guarantees the sampled diversity. Experiments show that
the ACL-DQN significantly improves the effectiveness and stability of dialogue
tasks with a statistically significant margin. Furthermore, the framework can
be further improved by equipping with different curriculum schedules, which
demonstrates that the framework has strong generalizability.
