  Building a dialogue system that can communicate naturally with humans is a
challenging yet interesting problem of agent-based computing. The rapid growth
in this area is usually hindered by the long-standing problem of data scarcity
as these systems are expected to learn syntax, grammar, decision making, and
reasoning from insufficient amounts of task-specific dataset. The recently
introduced pre-trained language models have the potential to address the issue
of data scarcity and bring considerable advantages by generating contextualized
word embeddings. These models are considered counterpart of ImageNet in NLP and
have demonstrated to capture different facets of language such as hierarchical
relations, long-term dependency, and sentiment. In this short survey paper, we
discuss the recent progress made in the field of pre-trained language models.
We also deliberate that how the strengths of these language models can be
leveraged in designing more engaging and more eloquent conversational agents.
This paper, therefore, intends to establish whether these pre-trained models
can overcome the challenges pertinent to dialogue systems, and how their
architecture could be exploited in order to overcome these challenges. Open
challenges in the field of dialogue systems have also been deliberated.
